Skip to content

fix(workflows): match overlay file extensions case-insensitively - #4531

Open
jawwad-ali wants to merge 2 commits into
github:mainfrom
jawwad-ali:fix/overlay-suffix-case-insensitive
Open

jawwad-ali wants to merge 2 commits into
github:mainfrom
jawwad-ali:fix/overlay-suffix-case-insensitive

Conversation

@jawwad-ali

@jawwad-ali jawwad-ali commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem

ProjectOverlaySource.collect matches the extension verbatim:

if not path.is_file() or path.suffix not in (".yml", ".yaml"):
    continue

So a hand-placed overlay named <id>.YML or <id>.Yaml is skipped and never applied — and nothing is reported to say the file was ignored. Overlay files are explicitly hand-authored (docs/reference/workflows.md documents the format and tells users to write them), so the extension casing is the author's choice, not something the tool generated.

Reproduction (before the fix)

Three overlays in one directory, differing only in extension case:

files on disk: ['Mixed.Yaml', 'UPPER.YML', 'lower.yml']
COLLECTED    : ['lower']
SKIPPED      : ['mixed', 'upper']

The author gets no error and no warning — the overlay simply has no effect.

Why this is the odd one out

Every other YAML discovery path in the package already lowercases before matching:

location form
workflows/engine.py (WorkflowEngine.load_workflow) path.suffix.lower() in (".yml", ".yaml")
workflows/command_add.py (two sites) dev_path.suffix.lower() / source_path.suffix.lower()
workflows/command_run.py source_path.suffix.lower() in (...)
workflows/overlay/layer_sources.py (ProjectOverlaySource.collect) path.suffix in (...) ← this PR
workflows/overlay/operations.py (_find_overlay_file) path.suffix in (...) ← this PR

Fix

Lowercase the suffix in both overlay paths, so the resolver and the management commands agree on which files are overlays:

if not path.is_file() or path.suffix.lower() not in (".yml", ".yaml"):
  • ProjectOverlaySource.collect — a <id>.YML overlay is now applied.
  • _find_overlay_file (added at @mnriem's request) — the same overlay is now reachable by workflow overlay enable / disable / remove. Fixing only the loader would have left it applied, but impossible to manage: every management command reported it "not found".

Verification

  • Loader: tests/workflows/test_overlay_layer_sources.py, parametrized over .YML, .Yaml, .YAML, .Yml. All 4 fail with the source reverted to main and pass with the fix. A companion test asserts that broadening the case did not broaden which extensions are accepted: notes.txt, backup.yml.bak, README.md and data.json are still skipped.
  • Management commands: one focused test per command in its mirrored suite (tests/specify_cli/workflows/overlay/test_command_disable.py, test_command_enable.py, test_command_remove.py), each parametrized over lint.YML / lint.Yaml. All 6 fail with the source reverted to main and pass with the fix.
  • Scoped regression over tests/workflows, tests/specify_cli/workflows/overlay and test_command_resolve.py: 235 passed. The 10 failures are the same pre-existing Windows symlink-privilege tests that fail on unmodified main; none are new.
  • uvx ruff@0.15.0 check src tests → clean.

Behaviour change, disclosed: files previously skipped now get read, and the overlay management commands now find them. If a project already holds a .YML/.YAML file in an overlay directory that is not a valid overlay manifest, resolution will now report it rather than ignoring it — which is the intended outcome, but it is a change for such a project.


Written with assistance from Claude Code. Bug found, reproduced, and verified by me on current main.

🤖 Generated with Claude Code

@jawwad-ali
jawwad-ali requested a review from mnriem as a code owner September 11, 2026 16:02
@mnriem mnriem added the triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review label Sep 12, 2026
@mnriem
mnriem requested a balanced review from Copilot September 16, 2026 11:54
@mnriem

mnriem commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Please include the corresponding case-insensitive check in _find_overlay_file() here. Otherwise these overlays become active while the management commands cannot locate them.

Add coverage showing that an uppercase-extension overlay can also be disabled, removed, and have its priority changed. This completes the same behavior fix rather than expanding scope.

Coordinate with #4141 through normal rebasing—whichever lands later should reconcile with the earlier change.

Drafted for @mnriem with assistance from GitHub Copilot (model: GPT-6 Astra; interactive comment drafting).

@mnriem mnriem added the author-awaiting Waiting on author response label Sep 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Overlay management commands still cannot find uppercase-extension overlays accepted by resolution.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates project overlay discovery to accept case-insensitive YAML extensions.

Changes:

  • Normalizes overlay suffixes before matching.
  • Adds regression and rejection tests.
File summaries
File Description
src/specify_cli/workflows/overlays/layer_sources.py Adds case-insensitive extension matching.
tests/workflows/test_overlay_layer_sources.py Tests mixed-case YAML and rejected extensions.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/specify_cli/workflows/overlay/layer_sources.py

@mnriem mnriem left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address Copilot feedback and resolve conflicts

@mnriem
mnriem self-requested a review September 25, 2026 12:52
@jawwad-ali
jawwad-ali force-pushed the fix/overlay-suffix-case-insensitive branch from c8ab0ac to 06f9932 Compare October 5, 2026 16:35
@jawwad-ali

Copy link
Copy Markdown
Contributor Author

@mnriem Conflicts resolved and feedback addressed (06f9932). Both halves are in: ProjectOverlaySource.collect in workflows/overlay/layer_sources.py and the matching case-insensitive check in _find_overlay_file() (workflows/overlay/operations.py) you asked for, so an uppercase-extension overlay that the resolver applies is also reachable by overlay enable/disable/remove.

tests/workflows/test_overlay_commands.py was split on main, so the management test now sits in tests/specify_cli/workflows/overlay/test_command_list.py beside the other _find_overlay_file tests. I also replaced a source comment that cited _commands.py line numbers, since that file no longer exists.

Verified: 6 new cases fail with both sources reverted to main and all pass with the fix.

Rebased as a single commit on current main; uvx ruff@0.15.0 check src tests is clean. Any remaining local failures are the pre-existing Windows symlink-privilege tests, which fail identically on unmodified main.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The new multi-command regression test is placed in the list command’s dedicated test suite, contrary to the repository’s mirrored test structure.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
Resolved since last review (1)

Comment thread tests/specify_cli/workflows/overlay/test_command_list.py Outdated
@mnriem

mnriem commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

jawwad-ali and others added 2 commits October 8, 2026 00:22
`ProjectOverlaySource.collect` matched `path.suffix` verbatim:

    if not path.is_file() or path.suffix not in (".yml", ".yaml"):
        continue

so a hand-placed overlay named `<id>.YML` or `<id>.Yaml` was skipped and never
applied, with nothing reported to say the file had been ignored. Overlay files
are explicitly hand-authored (docs/reference/workflows.md documents the format
and tells users to write them), so the casing is the author's choice.

Reproduced on main -- three overlays in one directory, differing only in
extension case:

    files on disk: ['Mixed.Yaml', 'UPPER.YML', 'lower.yml']
    COLLECTED    : ['lower']
    SKIPPED      : ['mixed', 'upper']

Every other YAML discovery path in the package already lowercases before
matching: engine.py:941, _commands.py:1325, :1894, :2128. This brings the
overlay loader in line with them.

Rebased onto current main (files moved in the workflow/bundler restructure).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The `_find_overlay_file` regression test drove `disable`, `enable` and
`remove` from `test_command_list.py`, contrary to the mirrored test layout in
design/cli.md ("Command-focused tests mirror the source command surface").
Split it into one focused test per command, in `test_command_disable.py`,
`test_command_enable.py` and `test_command_remove.py`, each parametrized over
`lint.YML` and `lint.Yaml`. `test_command_list.py` is back to its `main`
content. Test-only change; all six new cases fail with the source reverted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jawwad-ali
jawwad-ali force-pushed the fix/overlay-suffix-case-insensitive branch from 06f9932 to 324dfef Compare October 7, 2026 19:24
@jawwad-ali

Copy link
Copy Markdown
Contributor Author

@mnriem Copilot feedback addressed in 324dfef, and the thread is resolved. Per design/cli.md's mirrored test layout, the regression test now lives in the per-command suites (test_command_disable.py, test_command_enable.py, test_command_remove.py), one focused test each over lint.YML / lint.Yaml, and test_command_list.py is back to its main content. All 6 cases fail with the source reverted and pass with the fix; there are no new failures across the overlay suites, and ruff is clean. I also refreshed the PR description, which still said _find_overlay_file was out of scope.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The shared lookup change also affects add and set-priority, but those command paths lack regression coverage.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced

# lowercasing the suffix and this helper not, a ``<id>.YML`` overlay was
# ACTIVE during resolution yet reported "not found" by overlay
# enable / disable / remove -- applied, but impossible to manage.
if not path.is_file() or path.suffix.lower() not in (".yml", ".yaml"):
@mnriem

mnriem commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-awaiting Waiting on author response triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants