Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .ai/references/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Follow the style introduced in [#14113](https://github.com/huggingface/diffusers
- `HunyuanDiTAttentionPool` (`src/diffusers/models/embeddings.py`) shows the same failure without an MHA module: a plain `nn.Module` that hands its `q_proj` / `k_proj` / `v_proj` / `c_proj` weights to `torch.nn.functional.multi_head_attention_forward`, so all four projections stay offloaded rather than just one. `HunyuanDiT2DModel` opts out of group offloading entirely with `_supports_group_offloading = False`.
- Before adding a skip or an exclusion, confirm the failure still reproduces — several existing skips are stale, having outlived the upstream cause.
- **A migration that surfaces a `src/` gap marks the test `xfail`, it does not patch the pipeline.** Give the marker a module-level name and a `reason` naming the exact gap (`PNDM_*` in `tests/pipelines/pndm/test_pndm.py` is the worked example), and prefer `strict=True` so the marker reports XPASS — and gets deleted — the day the pipeline is fixed. Use `strict=False` only when one mark covers a group whose members do not all fail. Marking a whole test class keeps the mixin's own marks (`@is_memory`, `@require_accelerator`) intact; overriding individual inherited tests drops the decorators they were declared with, so re-declare those too.
- **`from_pipe` tests** (a pipeline that is a variant of an existing one — PAG, AnimateDiff, ...) compose the shared `FromPipeTesterMixin` (`tests/pipelines/testing_utils/from_pipe.py`, exported from `..testing_utils`) in their own test class. It derives the original pipeline from `pipeline_class.__name__`; set `original_pipeline_repo` on the test class to pull it from a repo other than the default for that class. The unittest-era `PipelineFromPipeTesterMixin` in `tests/pipelines/test_pipelines_common.py` is what it replaces.
- **`from_pipe` tests** (a pipeline that is a variant of an existing one — PAG, AnimateDiff, ...) compose the shared `FromPipeTesterMixin` (`tests/pipelines/testing_utils/from_pipe.py`, exported from `..testing_utils`) in their own test class. It derives the original pipeline from `pipeline_class.__name__`; set `original_pipeline_repo` on the test class to pull it from a repo other than the default for that class.
- **A hardware gap is a conditional skip, not an xfail.** When a test fails only because the runner's cuDNN build has no kernel for an op — `RuntimeError: GET was unable to find an engine to execute this computation`, as Sana's depthwise `Conv2d` hits in bfloat16 — wrap the call in `skip_if_no_cudnn_engine()` (`tests/testing_utils.py`). It skips on that error and re-raises every other `RuntimeError`, so the test still runs wherever the kernel exists.
- **PAG pipelines** also compose `PAGPipelineTesterMixin` (`tests/pipelines/pag/testing_utils.py`) in place of `PipelineTesterMixin`: it adds `test_pag_disable_enable` and `test_pag_inference` on top, driven by `base_pipeline_class` and the `pag_*` knobs on the test class. Keep `test_pag_applied_layers` per pipeline — which layers PAG resolves to is model-specific.
- **`encode_prompt` reading a component that isn't a text encoder or tokenizer?** `test_encode_prompt_works_in_isolation` rebuilds the pipeline with only the components whose names contain `text` or `tokenizer`. When `encode_prompt` also needs another one — a `processor` used for chat templating, say — list it in `text_stack_component_names` on the config class rather than re-implementing the test.
Expand Down
21 changes: 10 additions & 11 deletions .github/workflows/nightly_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:

- name: Pipeline Tests Artifacts
if: ${{ always() }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: test-pipelines.json
path: reports
Expand Down Expand Up @@ -108,7 +108,7 @@ jobs:
cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt
- name: Test suite reports artifacts
if: ${{ always() }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: pipeline_${{ matrix.module }}_test_reports
path: reports
Expand Down Expand Up @@ -180,7 +180,7 @@ jobs:

- name: Test suite reports artifacts
if: ${{ always() }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: torch_${{ matrix.module }}_cuda_test_reports
path: reports
Expand Down Expand Up @@ -225,7 +225,7 @@ jobs:

- name: Test suite reports artifacts
if: ${{ always() }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: torch_compile_test_reports
path: reports
Expand Down Expand Up @@ -279,7 +279,7 @@ jobs:
cat reports/tests_big_gpu_torch_cuda_failures_short.txt
- name: Test suite reports artifacts
if: ${{ always() }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: torch_cuda_big_gpu_test_reports
path: reports
Expand Down Expand Up @@ -323,7 +323,6 @@ jobs:
pytest -n 1 --max-worker-restart=0 --dist=loadfile \
--make-reports=tests_torch_minimum_version_cuda \
tests/models/test_modeling_common.py \
tests/pipelines/test_pipelines_common.py \

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer exists.

tests/pipelines/test_pipeline_utils.py \
tests/pipelines/test_pipelines.py \
tests/pipelines/test_pipelines_auto.py \
Expand All @@ -338,7 +337,7 @@ jobs:

- name: Test suite reports artifacts
if: ${{ always() }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: torch_minimum_version_cuda_test_reports
path: reports
Expand Down Expand Up @@ -410,7 +409,7 @@ jobs:
cat reports/tests_${{ matrix.config.backend }}_torch_cuda_failures_short.txt
- name: Test suite reports artifacts
if: ${{ always() }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: torch_cuda_${{ matrix.config.backend }}_reports
path: reports
Expand Down Expand Up @@ -463,7 +462,7 @@ jobs:
cat reports/tests_pipeline_level_quant_torch_cuda_failures_short.txt
- name: Test suite reports artifacts
if: ${{ always() }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: torch_cuda_pipeline_level_quant_reports
path: reports
Expand Down Expand Up @@ -496,7 +495,7 @@ jobs:
run: mkdir -p combined_reports

- name: Download all test reports
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
with:
path: artifacts

Expand All @@ -522,7 +521,7 @@ jobs:
cat $CONSOLIDATED_REPORT_PATH >> $GITHUB_STEP_SUMMARY

- name: Upload consolidated report
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: consolidated_test_report
path: ${{ env.CONSOLIDATED_REPORT_PATH }}
Expand Down
11 changes: 6 additions & 5 deletions .github/workflows/pr_tests_gpu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ on:
- "src/diffusers/loaders/lora_base.py"
- "src/diffusers/loaders/lora_pipeline.py"
- "src/diffusers/loaders/peft.py"
- "tests/pipelines/test_pipelines_common.py"
- "tests/pipelines/testing_utils/**.py"
- "tests/models/test_modeling_common.py"
- "tests/models/testing_utils/**.py"
Comment on lines -15 to +17

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To have parity in our CI when test changes happen.

- "examples/**/*.py"
workflow_dispatch:

Expand Down Expand Up @@ -93,7 +94,7 @@ jobs:
runs-on:
group: aws-general-8-plus
container:
image: diffusers/diffusers-pytorch-cpu
image: diffusers/diffusers-pytorch-cpu:latest
outputs:
pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }}
steps:
Expand Down Expand Up @@ -134,7 +135,7 @@ jobs:
runs-on:
group: aws-g4dn-2xlarge
container:
image: diffusers/diffusers-pytorch-cuda
image: diffusers/diffusers-pytorch-cuda:latest
options: --shm-size "16gb" --ipc host --gpus all
steps:
- name: Checkout diffusers
Expand Down Expand Up @@ -200,7 +201,7 @@ jobs:
runs-on:
group: aws-g4dn-2xlarge
container:
image: diffusers/diffusers-pytorch-cuda
image: diffusers/diffusers-pytorch-cuda:latest
options: --shm-size "16gb" --ipc host --gpus all
defaults:
run:
Expand Down Expand Up @@ -272,7 +273,7 @@ jobs:
group: aws-g4dn-2xlarge

container:
image: diffusers/diffusers-pytorch-cuda
image: diffusers/diffusers-pytorch-cuda:latest
options: --gpus all --shm-size "16gb" --ipc host
steps:
- name: Checkout diffusers
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/release_tests_fast.yml
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,6 @@ jobs:
pytest -n 1 --max-worker-restart=0 --dist=loadfile \
--make-reports=tests_torch_minimum_cuda \
tests/models/test_modeling_common.py \
tests/pipelines/test_pipelines_common.py \

@sayakpaul sayakpaul Sep 3, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer exists.

tests/pipelines/test_pipeline_utils.py \
tests/pipelines/test_pipelines.py \
tests/pipelines/test_pipelines_auto.py \
Expand Down
2 changes: 1 addition & 1 deletion tests/models/transformers/test_models_transformer_flux.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@


# TODO: This standalone function maintains backward compatibility with pipeline tests
# (tests/pipelines/test_pipelines_common.py) and will be refactored.
# (tests/pipelines/testing_utils/ip_adapter.py) and will be refactored.
def create_flux_ip_adapter_state_dict(model) -> dict[str, dict[str, Any]]:
"""Create a dummy IP Adapter state dict for Flux transformer testing."""
ip_cross_attn_state_dict = {}
Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/deepfloyd_if/test_if.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@
slow,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
PipelineOffloadTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/deepfloyd_if/test_if_img2img.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@
slow,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
PipelineOffloadTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@
slow,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
PipelineOffloadTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/deepfloyd_if/test_if_inpainting.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@
slow,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
PipelineOffloadTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@
slow,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
PipelineOffloadTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/deepfloyd_if/test_if_superresolution.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@
slow,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
PipelineOffloadTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/kandinsky/test_kandinsky.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,11 @@
slow,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
MemoryTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/kandinsky/test_kandinsky_img2img.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,11 @@
slow,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
MemoryTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/kandinsky/test_kandinsky_inpaint.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@
require_torch_accelerator,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
MemoryTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/shap_e/test_shap_e.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,11 @@
require_torch_accelerator,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
MemoryTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
2 changes: 1 addition & 1 deletion tests/pipelines/shap_e/test_shap_e_img2img.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@
require_torch_accelerator,
torch_device,
)
from ..test_pipelines_common import assert_mean_pixel_difference
from ..testing_utils import (
BasePipelineTesterConfig,
MemoryTesterMixin,
PipelineTesterMixin,
assert_mean_pixel_difference,
)


Expand Down
Loading
Loading