Skip to content

perf(search): adopt Get Filtered Team Tasks instead of crawling every list - #48

Open
timimsms wants to merge 3 commits into
mainfrom
feat/filtered-team-tasks
Open

timimsms wants to merge 3 commits into
mainfrom
feat/filtered-team-tasks

Conversation

@timimsms

Copy link
Copy Markdown
Owner

Summary

Fixes #25. task search and export tasks walked workspace → spaces → folders → lists and called GetTasks per list. Three problems, all of which this replaces with one server-side query per workspace against GET /team/{team_id}/task:

  1. One request per list under a 100 req/min limit.
  2. Silent truncation — only page 0 was fetched per list, so any list past 100 tasks was quietly cut off.
  3. Silently partial exportsexport.go discarded GetFolders/GetLists errors (folders, _ := …), so whole subtrees could vanish and the export still reported success.

Measured on a small real workspace, task search: 3.74s → 1.14s. The gap widens with list count, since the old cost was linear in lists and the new one is linear in pages of results.

Truncation is now reported, not hidden

Pagination is bounded by DefaultMaxTaskPages and returns Truncated when it stops early, so a capped result is never presented as complete — that being the actual bug in this issue. Both commands print a warning when it happens.

The paging loop is extracted as paginate so termination and truncation are testable without a network round trip.

Behaviour is deliberately unchanged by default

The team endpoint can return closed tasks and subtasks, which the per-list crawl never did. Turning that on by default would have quietly changed what search returns — on this workspace, search cu went from 2 results to 4, the extra two being subtasks. That is a better answer, but it is not what this issue asked for, so:

  • defaults mirror the old crawl (open tasks, no subtasks), and I verified default results match v0.2.1 exactly on a real workspace;
  • the fuller set is reachable via new --include-closed and --subtasks flags on both commands.

--space given as a name still works: tasks carry only a space id, so the name is resolved to an id first rather than dropping the capability.

New: reverse lookup

Client.FindTasksByCustomField and cu field find — from a value stored in a custom field back to the task holding it. The crawl could not do this at any acceptable cost; the endpoint filters custom fields server-side.

$ cu field find Repo https://github.com/timimsms/cu
id          name        status      list          url
86dxbeqyt   cu          in review   R&D Projects  https://app.clickup.com/t/86dxbeqyt

A field id works anywhere; a field name is resolved against --list or the default list, since fields are only enumerable per list. No match exits 0 with an empty result rather than erroring.

Tests

TestPaginate covers stopping on a short page, a single short page, an empty first page, the exactly-full-last-page case (which costs one extra request to discover), hitting the cap and reporting truncation, and errors aborting rather than returning a partial set. TestTeamTaskQueryToSDK covers filter pass-through and omission of empty filters.

Checklist

  • ./scripts/ci.sh passes locally — except errcheck, which reports the same pre-existing findings on main, none in files this PR touches
  • Commit messages use conventional prefixes
  • CLI docs regenerated (cu docs markdown --dir docs/site/commands) and cu field find added to the mkdocs nav
  • Docs updated if user-facing behavior changed — README feature list and examples

… list

`task search` and `export tasks` walked workspace → spaces → folders → lists and
called GetTasks per list. That cost one request per list under a 100 req/min
limit, and since only page 0 was ever fetched, any list past 100 tasks was
silently cut off. export also discarded GetFolders/GetLists errors, so it could
omit whole subtrees and still report success.

Both now issue one server-side query per workspace against
GET /team/{id}/task, paged to exhaustion. Against a small real workspace this
takes search from 3.7s to 1.1s; the gap widens with list count.

Pagination is bounded by DefaultMaxTaskPages and reports Truncated when it stops
early, so a capped result is never presented as a complete one — the failure
mode this issue is about. The paging loop is extracted as `paginate` so
termination and truncation are testable without a network round trip.

Search and export defaults deliberately mirror the old crawl (open tasks, no
subtasks) to keep this a performance fix; the endpoint makes the fuller set
reachable via new --include-closed and --subtasks flags. Verified that default
results match v0.2.1 exactly on a real workspace.

Adds Client.FindTasksByCustomField and `cu field find`, the reverse lookup from
a value stored in a custom field back to the task holding it, which the crawl
could not do at any acceptable cost.

Fixes #25

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014aqbmccWm1tqttmBUCR5rv
ClickUp: 86dxbeqyt
These four files were swept into the previous commit by a broad `git add` while
another change was in progress in the same working tree. They are unrelated to
the search/export work this branch is about, and that commit's message does not
mention them — so a change to what `cu config get` prints, and a new write guard
on .cu.yml, were riding along undisclosed on a PR titled as a performance fix.

They now live on their own in #50, against a clean main, where the security
surface gets reviewed on its own terms. Nothing here depended on them: no code
on this branch references GlobalConfigPath, stripCredentials, RedactedValue or
IsCredentialKey, and the suite passes without them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZEGsLHBQ2GzP6v48i4hXz
@timimsms

Copy link
Copy Markdown
Owner Author

Note: four files removed from this branch

Heads up if you are working on this branch — I pushed a commit removing four files that did not belong to it.

internal/config/config.go, internal/config/config_test.go, internal/cmd/config.go and internal/cmd/config_test.go were swept into ac05574 by a broad git add, while an unrelated change was uncommitted in the same working tree. The result was that a change to what cu config get prints, plus a new write guard on .cu.yml, arrived on a PR whose message describes only the search/export performance work — so a reviewer reading that message had no reason to look for a security-surface change.

They now live on their own in #50, against a clean main.

Nothing on this branch depended on them: no code here references GlobalConfigPath, stripCredentials, RedactedValue or IsCredentialKey, and go build, go vet and the full suite pass without them. I verified the branch head was still ac05574 immediately before pushing, so nothing of yours was overwritten.

The search/export work itself is untouched — I have not reviewed it yet, and will do that separately.

Brings in #50 and #51. One conflict, in the export flag block, where this
branch's --include-closed/--subtasks met #51 moving the destination file off
-o/--output onto --file/-F. Both sides kept: the new filters stay, and -o is
left to the global format flag as #51 intends.

Resolved as a merge rather than a rebase because another session owns this
branch; force-pushing rewritten history over it is the one operation that could
destroy work in progress.

Docs regenerated for the combined flag set — the auto-merged page still
advertised -o as the output file, which is no longer true.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ZEGsLHBQ2GzP6v48i4hXz
@timimsms

Copy link
Copy Markdown
Owner Author

Updated against main — one conflict, resolved

Brought main in so this reviews against what would actually merge. #50 and #51 had landed since ac05574, and #51 touches export.go.

One conflict, in the export flag block — exactly where the two changes met:

<<<<<<< HEAD
	exportTasksCmd.Flags().Bool("include-closed", ...)
	exportTasksCmd.Flags().Bool("subtasks", ...)
	exportTasksCmd.Flags().StringP("output", "o", "", "Output file (default: stdout)")
=======
	exportTasksCmd.Flags().StringP("file", "F", "", "Write to a file instead of stdout")
>>>>>>> origin/main

Resolved keeping both sides: this branch's two new filters stay, and the destination file remains on --file/-F with -o left to the global format flag, per #51. The rest of export.go auto-merged so that the run body reads --file for the destination and resolveExportFormat for the format — I checked that rather than assuming it.

Docs needed regenerating. The auto-merged cu_export_tasks.md still advertised -o as the output file, which stopped being true in #51. The page now shows -F, --file locally and -o under Options inherited as the format flag, alongside --include-closed and --subtasks. Datestamp-only pages reverted, per the convention from #47.

Verified on the merged result, not just built: the combined flag surface is correct, and #51's behavior survives intact — -o tasks.csv still errors with the --file hint rather than writing a junk file, and --format csv -o json still refuses to guess. Full suite, go vet and gofmt clean. CI is 16/16.

The branch now differs from main only by this PR's own work — I confirmed the config files are byte-identical to main, so #50 arrived through the merge rather than resurrecting the copies that were split out earlier.

Why a merge and not a rebase

Another session owns this branch. Force-pushing rewritten history over it is the one operation that could destroy work in progress, so this is an ordinary merge commit — which the repo already does elsewhere (Merge branch 'main' into docs/public-launch-prep). Say the word if you want it linearised once that session is done and I'll rebase properly.

Still unreviewed by me — that's next.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

search/export: unbounded N+1 workspace crawl with silent truncation; adopt Get Filtered Team Tasks

1 participant