Skip to content

Fix github-discussion-query payload scaling by eliminating jq argument growth - #55560

Merged
pelikhan merged 3 commits into
mainfrom
copilot/deep-report-fix-argument-list-limit
Aug 25, 2026
Merged

Fix github-discussion-query payload scaling by eliminating jq argument growth#55560
pelikhan merged 3 commits into
mainfrom
copilot/deep-report-fix-argument-list-limit

Conversation

Copilot AI commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

github-discussion-query silently degraded coverage for higher --limit values because discussion payloads were passed through jq via shell arguments/here-strings, eventually hitting OS argument-length limits. This capped effective scan depth and caused daily report cross-check gaps.

  • Pagination/data path hardening

    • Replaced in-memory JSON accumulation (--argjson all "$OUTPUT") with temp-file-backed merges (jq -s) so payload size no longer depends on shell argument limits.
    • Kept existing since pagination semantics and early-stop behavior while removing large argument transport from the hot path.
  • Filter/output execution path

    • Switched jq filter execution from here-string input to file input:
      • before: jq "$JQ_FILTER" <<< "$OUTPUT"
      • after: jq "$JQ_FILTER" "$OUTPUT_FILE"
    • Updated item count and byte-size derivation to read from file-backed output while preserving prior metadata behavior.
  • Temp-file lifecycle safety

    • Added explicit temp-file tracking and trap cleanup for all intermediates used during page extraction, merge, and final projection.
# Before (argument-growth prone)
OUTPUT=$(jq -cn --argjson all "$OUTPUT" --argjson page "$PAGE_NODES" '$all + $page')
jq "$JQ_FILTER" <<< "$OUTPUT"

# After (payload-size resilient)
jq -s '.[0] + .[1]' "$OUTPUT_FILE" "$PAGE_NODES_FILE" > "$MERGED_OUTPUT_FILE"
jq "$JQ_FILTER" "$OUTPUT_FILE"

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for tackling the discussion-query pagination limit! This fix — switching from <<< here-strings to streaming via temp files — addresses the real bottleneck blocking the Daily Regulatory Report from scanning beyond ~12 items.

Since this is still a draft, here's what would help get it across the finish line:

  • Add tests — The PR checklist notes "[ ] Add or update focused tests/fixtures for high-limit discussion queries". Specifically, add fixtures for queries with --limit values well above 12 (e.g., 50+) to confirm the argument-length issue is truly resolved and won't regress.
  • Complete final checks — The checklist also flags "[ ] Run final checks (code review, CodeQL, secrets scan)". Once tests pass, push those final validation runs.

The core refactor looks solid — swapping to temp files + jq -s file-to-file operations is the right approach to sidestep shell argument limits entirely.

Once the tests are in place and the checklist is complete, this should be ready for merge!

Generated by ✅ Contribution Check · copilot · auto · 46.5 AIC · ⌖ 6.86 AIC · ⊞ 9.3K ·

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix 'Argument list too long' limit in github-discussion-query tool Fix github-discussion-query payload scaling by eliminating jq argument growth Aug 25, 2026
Copilot AI requested a review from pelikhan August 25, 2026 01:20
@pelikhan
pelikhan marked this pull request as ready for review August 25, 2026 02:12
Copilot AI balanced review requested due to automatic review settings August 25, 2026 02:12
@pelikhan
pelikhan merged commit e5f3bc9 into main Aug 25, 2026
2 checks passed
@pelikhan
pelikhan deleted the copilot/deep-report-fix-argument-list-limit branch August 25, 2026 02:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Reworks github-discussion-query to use temporary files, avoiding shell argument-length limits for large result sets.

Changes:

  • Uses file-backed page merging and jq filtering.
  • Adds automatic temporary-file cleanup.
  • Regenerates all importing workflow lock files.
Show a summary per file
File Description
.github/workflows/shared/github-queries-mcp-script.md Implements scalable file-backed discussion processing.
.github/workflows/daily-regulatory.lock.yml Regenerates the imported query script.
.github/workflows/daily-performance-summary.lock.yml Regenerates the imported query script.
.github/workflows/smoke-copilot.lock.yml Regenerates the imported query script.
.github/workflows/smoke-copilot-arm.lock.yml Regenerates the imported query script.
.github/workflows/smoke-copilot-aoai-entra.lock.yml Regenerates the imported query script.
.github/workflows/smoke-copilot-aoai-apikey.lock.yml Regenerates the imported query script.

Review details

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 7/8 changed files
  • Comments generated: 0
  • Review effort level: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.87.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[deep-report] Fix "Argument list too long" limit (~12 items) in github-discussion-query mcp-script tool

3 participants