Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions .ai/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@
- Local Claude Code agents: run `make claude` after cloning to wire the [skills](#skills) under `.claude/`.
- Local OpenAI Codex agents: run `make codex` after cloning to wire the [skills](#skills) under `.agents/`.

Files under `.ai/` are read through several paths — this file is also `CLAUDE.md` and
`AGENTS.md` at the repo root, and `.ai/skills/` is also `.claude/skills/` and `.agents/skills/`.
Write cross-file references as paths from the repo root (`` `.ai/models.md` ``), not as
relative links: `../../models.md` points at `.ai/` or `.claude/` depending on which path the
reader arrived through.

## Coding style

Strive to write code as simple and explicit as possible.
Expand All @@ -27,20 +33,20 @@ Strive to write code as simple and explicit as possible.

## Reference guides

- **Models** — see [models.md](models.md) for model conventions, attention pattern, implementation rules, dependencies, and gotchas. For adding or converting a model, use the [model-integration](./skills/model-integration/SKILL.md) skill.
- **Pipelines** — see [pipelines.md](pipelines.md) for pipeline conventions, patterns, and gotchas.
- **Modular pipelines** — see [modular.md](modular.md) for modular pipeline conventions, patterns, and gotchas.
- **Tests** — see [testing.md](testing.md) for test conventions: required test layers, tester mixins, and dummy-component rules.
- **Models** — see `.ai/models.md` for model conventions, attention pattern, implementation rules, dependencies, and gotchas. For adding or converting a model, use the `model-integration` skill.
- **Pipelines** — see `.ai/pipelines.md` for pipeline conventions, patterns, and gotchas.
- **Modular pipelines** — see `.ai/modular.md` for modular pipeline conventions, patterns, and gotchas.
- **Tests** — see `.ai/testing.md` for test conventions: required test layers, tester mixins, and dummy-component rules.

## Skills

Task-specific guides live in `.ai/skills/` and are loaded on demand by AI agents. Available skills include:

- [model-integration](./skills/model-integration/SKILL.md) (adding/converting pipelines)
- [custom-blocks](./skills/custom-blocks/SKILL.md) (packaging a `ModularPipelineBlocks` subclass for the Hub)
- [diffusers-cli](./skills/diffusers-cli/SKILL.md) (running pipelines, inspecting schemas, and using the Diffusers CLI)
- [self-review](./skills/self-review/SKILL.md) (pre-PR self-review against the project rules)
- `model-integration` (`.ai/skills/model-integration/SKILL.md`) — adding/converting pipelines
- `custom-blocks` (`.ai/skills/custom-blocks/SKILL.md`) — packaging a `ModularPipelineBlocks` subclass for the Hub
- `diffusers-cli` (`.ai/skills/diffusers-cli/SKILL.md`) — running pipelines, inspecting schemas, and using the Diffusers CLI
- `self-review` (`.ai/skills/self-review/SKILL.md`) — pre-PR self-review against the project rules

## Self-review before a PR

Before opening a PR, run self-review against [review-rules.md](review-rules.md). The [self-review skill](skills/self-review/SKILL.md) runs this as the same pass the `@claude` CI reviewer uses. Share the final report on the PR (description or comment) — see the skill for details.
Before opening a PR, run self-review against `.ai/review-rules.md`. The `self-review` skill (`.ai/skills/self-review/SKILL.md`) runs this as the same pass the `@claude` CI reviewer uses. Share the final report on the PR (description or comment) — see the skill for details.
20 changes: 10 additions & 10 deletions .ai/skills/model-integration/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
name: integrating-models
name: model-integration
description: >
Use when adding a new model or pipeline to diffusers, setting up file
structure for a new model, converting a pipeline to modular format, or
Expand All @@ -16,7 +16,7 @@ Before writing any code, gather info in this order:

1. **Reference repo** — ask for the github link. If they've already set it up locally, ask for the path. Otherwise, ask what setup steps are needed (install deps, download checkpoints, set env vars, etc.) and run through them before proceeding.
2. **Inference script** — ask for a runnable end-to-end script for a basic workflow first (e.g. T2V). Then ask what other workflows they want to support (I2V, V2V, etc.) and agree on the full implementation order together.
3. **Standard vs modular** — **default to modular.** [Modular Diffusers](../../modular.md) is the preferred implementation for new pipelines; the standard `DiffusionPipeline` is still supported but no longer the default. We prefer modular especially for models that don't fit a fixed task-based structure (modality baked into the checkpoint) or that are actively evolving.
3. **Standard vs modular** — **default to modular.** Modular Diffusers (`.ai/modular.md`) is the preferred implementation for new pipelines; the standard `DiffusionPipeline` is still supported but no longer the default. We prefer modular especially for models that don't fit a fixed task-based structure (modality baked into the checkpoint) or that are actively evolving.

Ask step 3 as an `AskUserQuestion`, with modular marked as the recommended default.

Expand All @@ -29,15 +29,15 @@ Then work through the **Integration checklist** below
A pipeline in Diffusers (be it standard or modular) will have multiple components. These components can be models, schedulers, processors, etc.

- [ ] **Transformer model**
- [ ] Implement the model with `from_pretrained` support (conventions: [models.md](../../models.md))
- [ ] Implement the model with `from_pretrained` support (conventions: `.ai/models.md`)
- [ ] Convert weights (see **Weight / Checkpoint Conversion**)
- [ ] Parity test against the reference (internal, not shipped — see **Model parity test**)
- [ ] Register in the relevant `__init__.py` files (lazy imports)
- [ ] Model-level tests (see **Testing**)
- [ ] **VAE** (if applicable) — reuse an existing `AutoencoderKL*` if possible; if a new one is needed, follow the same sub-steps as the transformer
- [ ] **Scheduler** — reuse an existing scheduler, or add a custom one
- [ ] **Pipeline**
- [ ] Implement the pipeline — see [modular.md](../../modular.md) for modular pipeline, or [pipelines.md](../../pipelines.md) for standard pipeline
- [ ] Implement the pipeline — see `.ai/modular.md` for modular pipeline, or `.ai/pipelines.md` for standard pipeline
- [ ] Add a LoRA mixin if applicable
- [ ] Register in the relevant `__init__.py` files (lazy imports)
- [ ] Pipeline-level tests (see **Testing**)
Expand All @@ -53,8 +53,8 @@ src/diffusers/
models/transformers/transformer_<model>.py # the model (or models/autoencoders/, models/unets/)
schedulers/scheduling_<model>.py # only if a custom scheduler is needed
loaders/lora_pipeline.py # LoRA mixin — add to the existing file
pipelines/<model>/ # standard pipeline — see pipelines.md
modular_pipelines/<model>/ # modular pipeline — see modular.md
pipelines/<model>/ # standard pipeline — see .ai/pipelines.md
modular_pipelines/<model>/ # modular pipeline — see .ai/modular.md
tests/
models/transformers/test_models_transformer_<model>.py
pipelines/<model>/test_<model>.py
Expand All @@ -66,7 +66,7 @@ docs/source/en/

## Model integration specific rules

**Match the reference's numerical logic.** Restructuring code to fit diffusers APIs (`ModelMixin`, `ConfigMixin`, blocks for modular, etc.) is expected, and required diffusers conventions (e.g. the attention pattern in [models.md](../../models.md)) take precedence. Beyond those, keep the actual computation as close to the reference as possible — don't reorder operations, change the math, or rename internals for aesthetics, even if it looks unclean. Small deviations make output mismatches very hard to track down.
**Match the reference's numerical logic.** Restructuring code to fit diffusers APIs (`ModelMixin`, `ConfigMixin`, blocks for modular, etc.) is expected, and required diffusers conventions (e.g. the attention pattern in `.ai/models.md`) take precedence. Beyond those, keep the actual computation as close to the reference as possible — don't reorder operations, change the math, or rename internals for aesthetics, even if it looks unclean. Small deviations make output mismatches very hard to track down.

## Weight / Checkpoint Conversion

Expand All @@ -76,7 +76,7 @@ Convert the original checkpoint into diffusers format with a standalone script u
2. Instantiate the diffusers model from its config and load the converted state dict.
3. `save_pretrained(...)` to a local path, then load it back with `from_pretrained` to confirm it round-trips.

All weights load through the standard paths — `from_pretrained`, or `from_single_file` (add `FromSingleFileMixin` + a weight-mapping) for an original-format single checkpoint. No custom `from_pretrained`, no manual runtime loading. See the loading rule in [models.md](../../models.md).
All weights load through the standard paths — `from_pretrained`, or `from_single_file` (add `FromSingleFileMixin` + a weight-mapping) for an original-format single checkpoint. No custom `from_pretrained`, no manual runtime loading. See the loading rule in `.ai/models.md`.

Common conversion patterns to watch for model-level components:
- Fused QKV weights that need splitting into separate Q, K, V
Expand All @@ -87,13 +87,13 @@ Common conversion patterns to watch for model-level components:

## Testing

Two test layers must be added for any new pipeline: pipeline-level tests, and (if a new model is introduced) model-level tests. Conventions for both layers — file locations, tester mixins, dummy-component rules — live in [testing.md](../../testing.md); follow it when writing the tests.
Two test layers must be added for any new pipeline: pipeline-level tests, and (if a new model is introduced) model-level tests. Conventions for both layers — file locations, tester mixins, dummy-component rules — live in `.ai/testing.md`; follow it when writing the tests.

## Model parity test

Confirm the diffusers implementation matches the reference. Test each component on **CPU/float32** with a strict tolerance (`max_diff < 1e-3`), comparing the **freshly converted** weights against the reference in a single script — both sides side by side, nothing saved to disk in between. See [pitfalls.md](pitfalls.md) for the common sources of numerical discrepancy.

This is an **internal verification tool for integration — it should not be shipped in the PR** (it imports the reference repo). The tests that ship with the PR are the model-level and pipeline-level tests in [testing.md](../../testing.md).
This is an **internal verification tool for integration — it should not be shipped in the PR** (it imports the reference repo). The tests that ship with the PR are the model-level and pipeline-level tests in `.ai/testing.md`.

The example below is schematic (placeholder names). `ReferenceModel` is the component **imported from the original repo**, and `convert_my_component` is **the same conversion function you wrote for the conversion script for the component**. You should make sure both load the *same* checkpoint weights and run the *same* input, so any difference is a conversion or implementation bug — not a difference in inputs.

Expand Down
Loading