Skip to content

Align Visualized-BGE multimodal attention masks - #1598

Open
zjn20030811 wants to merge 1 commit into
FlagOpen:masterfrom
zjn20030811:nan/visual-bge-mask-alignment
Open

Align Visualized-BGE multimodal attention masks#1598
zjn20030811 wants to merge 1 commit into
FlagOpen:masterfrom
zjn20030811:nan/visual-bge-mask-alignment

Conversation

@zjn20030811

Copy link
Copy Markdown

Summary

  • align Visualized-BGE multimodal attention masks with the [CLS, image tokens, text] embedding layout
  • preserve the prompt mask dtype/device and validate mask/length inputs
  • add regression tests for padded prompts, boolean masks, and empty image-token sequences

Problem

encode_mm prepends image tokens after the prompt CLS embedding and removes the original prompt CLS from the remaining text sequence. Its mask was built as [image mask, full prompt mask], shifting every mask entry by one position. As a result, padding could be attended to while valid text or image positions were masked, and mean pooling used the wrong validity positions.

The new helper builds [prompt CLS mask, image mask, prompt remainder mask], exactly matching the sequence passed to the encoder. Existing model weights and APIs are unchanged.

Validation

  • python -m pytest -q tests/test_visual_bge_attention_mask.py (4 passed)
  • python -m compileall -q research/visual_bge/visual_bge/modeling.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant