Stop passing fps to the transformer in Cosmos2VideoToWorldPipeline - #14940
Open
UniversePeak wants to merge 1 commit into
Open
UniversePeak wants to merge 1 commit into
UniversePeak wants to merge 1 commit into
Conversation
The released Cosmos-Predict2 checkpoints ship with rope_enable_fps_modulation=False, so natively they use raw integer temporal RoPE positions and ignore FPS entirely. The diffusers port of Cosmos2VideoToWorldPipeline always passed fps (default 16) to CosmosTransformer3DModel, whose CosmosRotaryPosEmbed scales temporal positions by base_fps / fps = 24 / 16 = 1.5x for every default run, diverging from the reference implementation and making the playback fps change the generated content. Cosmos2TextToImagePipeline and the Cosmos 2.5 pipelines already pass no fps, and the transformer class is shared with Cosmos-Predict1, whose pipelines still pass it, so the pipeline is the right place to encode this difference. fps stays a playback-only argument of the pipeline.
github-actions
Bot
added
tests
pipelines
fixes-issue
size/S
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #14768 (the second option suggested there).
The released Cosmos-Predict2 checkpoints ship with
rope_enable_fps_modulation=False(config_video2world.py, same in the 14B and text-to-image nets), so natively they use raw integer temporal RoPE positions and ignore FPS entirely.Cosmos2VideoToWorldPipelinewas the only Predict2-family pipeline still passingfpsinto the transformer, whoseCosmosRotaryPosEmbedscales temporal positions bybase_fps / fps = 24 / 16 = 1.5xon every default run, and the playbackfpsargument changed the generated content.This PR stops passing
fpsto the transformer and documentsfpsas a playback-only argument. I went with this instead of portingenable_fps_modulationintoCosmosTransformer3DModelbecause:nvidia/Cosmos-Predict2-*config can contain a key that does not exist in the class today, soregister_to_configwould hand every released checkpoint the class default; the flag would carry no per-checkpoint information.CosmosTransformer3DModelis shared with Cosmos-Predict1, which always applies the modulation (its pipelines passfps). A default ofFalsewould silently break Predict1 andTruekeeps this bug, so the pipeline is where the family difference belongs.Note that this changes the output of every default
Cosmos2VideoToWorldPipelinerun: on this branchfps=16reproduces whatfps=24produced before, i.e. the un-modulated temporal RoPE the checkpoints were trained with.Tests
test_fps_does_not_change_generation: the pipeline output forfps16, 24 and 30 must be identical. On main it fails with max abs diff 0.0824; with this change it passes.pytest tests/pipelines/cosmos/test_cosmos2_video2world.py tests/models/transformers/test_models_transformer_cosmos.py tests/pipelines/cosmos/test_cosmos.py tests/pipelines/cosmos/test_cosmos_video2world.py— 98 passed, 78 skipped on CPU.ruff checkandruff format --checkclean on the touched files.