Skip to content

Migrate skill aggregate to skills/ and unblock NVSkills evaluations - #27

Merged
ohadmo merged 11 commits into
mainfrom
omosafi/evals
Aug 25, 2026
Merged

ohadmo merged 11 commits into
mainfrom
omosafi/evals

Conversation

@ohadmo

@ohadmo ohadmo commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Migrates the generated BioNeMo Agent Toolkit aggregate from
plugins/bionemo-agent-toolkit/ to skills/bionemo-agent-toolkit/ so it is
discovered through the current NVSkills validation path.

The legacy plugins/ path remains as a temporary symlink to skills/ for
compatibility with existing consumers.

Changes

  • Move the generated bionemo-agent-toolkit aggregate under skills/.
  • Keep plugins -> skills as a temporary compatibility symlink.
  • Update the Codex and Claude Code marketplace manifests to use the canonical
    skills/bionemo-agent-toolkit/ path.
  • Update documentation, GitHub workflow path filters, and skill synchronization
    workflows for the new layout.
  • Update plugin_sync.py to:
    • validate the new aggregate location;
    • verify the compatibility symlink;
    • retain evaluation datasets in the aggregate;
    • preserve NVSkills-generated BENCHMARK.md, skill-card.md, and
      skill.oms.sig artifacts during regeneration;
    • continue enforcing source coverage and freshness.
  • Normalize the nvMolKit catalog directory from nvMolKit to
    nvmolkit-usage, matching the skill name.
  • Add the returned NVSkills benchmark, skill card, and signature for
    nvmolkit-usage.
  • Retrigger NVSkills validation for the remaining catalog skills.

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@ohadmo

ohadmo commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

@ohadmo ohadmo self-assigned this Aug 13, 2026
@greptile-apps

greptile-apps Bot commented Aug 13, 2026

Copy link
Copy Markdown

Greptile Summary

The PR resolves evaluation and synchronization failures by renaming the nvMolKit skill consistently, retaining generated evaluation/signing artifacts during payload rebuilds, and refreshing generated plugin content.

  • Renames nvMolKit catalog and payload paths to nvmolkit-usage.
  • Preserves payload-only benchmark, skill-card, and signature artifacts during plugin synchronization.
  • Adds evaluation evidence and a root skills link for plugin discovery.
  • Touches skill documents so NVSkills CI revalidates them.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename Overview
scripts/plugin_sync.py Excludes generated payload-only artifacts from freshness comparisons and preserves them across payload rebuilds.
components.d/nvmolkit.yml Updates the nvMolKit catalog destination to the renamed library-skills/nvmolkit-usage directory.
plugins.d/bionemo-agent-toolkit.yml Updates plugin inclusion to reference the renamed nvMolKit source path.
skills.sh.json Updates the published CUDA-X skill identifier from nvMolKit to nvmolkit-usage.
plugins/bionemo-agent-toolkit/skills/nvmolkit-usage/BENCHMARK.md Adds publication-oriented evaluation results and metadata for the renamed skill.

Reviews (5): Last reviewed commit: "add symlink" | Re-trigger Greptile

svc-nvskills-signing and others added 2 commits August 13, 2026 14:23
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@ohadmo

ohadmo commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

The signing service commits skill.oms.sig, BENCHMARK.md, and skill-card.md
into the plugin payload only (never authored in source). plugin_sync --check
flagged these as "extra in payload" drift, and --write would delete them on
the rmtree+copytree rebuild — silently dropping a committed signature.

Treat them as generated, payload-only artifacts: skip skill-root
skill.oms.sig / BENCHMARK.md / skill-card.md in the freshness comparison,
and preserve them across a --write rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@ohadmo

ohadmo commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@ohadmo

ohadmo commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@greptile-apps

greptile-apps Bot commented Aug 14, 2026

Copy link
Copy Markdown

Too many files changed for review (308 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.

@ohadmo

ohadmo commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@ohadmo ohadmo changed the title Resolve Evals failures Migrate skill aggregate to skills/ and unblock NVSkills evaluations Aug 14, 2026
@ohadmo

ohadmo commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@ohadmo

ohadmo commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

ohadmo added 2 commits August 21, 2026 11:34
Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@ohadmo

ohadmo commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
@ohadmo

ohadmo commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

1 similar comment
@ohadmo

ohadmo commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

@ohadmo
ohadmo merged commit 9fa5b67 into main Aug 25, 2026
3 of 6 checks passed
ohadmo added a commit that referenced this pull request Aug 25, 2026
)

* Fix nvmolkit

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

* Attach NVSkills validation signatures

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>

* TO DROP

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

* Reconcile plugin_sync with NVSkills signing artifacts

The signing service commits skill.oms.sig, BENCHMARK.md, and skill-card.md
into the plugin payload only (never authored in source). plugin_sync --check
flagged these as "extra in payload" drift, and --write would delete them on
the rmtree+copytree rebuild — silently dropping a committed signature.

Treat them as generated, payload-only artifacts: skip skill-root
skill.oms.sig / BENCHMARK.md / skill-card.md in the freshness comparison,
and preserve them across a --write rebuild.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

* add symlink

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

* skills folder renaming

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

* rename plugin/ folder to skills/

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

* reword binder-design eval prompts to avoid model safety guardrails

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

* resolve merge conflicts

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

* Suppress evo2-nim Tier 1 SQP-2 high via audited SkillSpector baseline

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>

---------

Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Ohad Mosafi <omosafi@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants