Skip to content

Avoid listing models for explicit text model IDs - #231

Open
chubes4 wants to merge 4 commits into
WordPress:trunkfrom
chubes4:explicit-model-no-list
Open

Avoid listing models for explicit text model IDs#231
chubes4 wants to merge 4 commits into
WordPress:trunkfrom
chubes4:explicit-model-no-list

Conversation

@chubes4

@chubes4 chubes4 commented May 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a provider-overridable path for explicit model IDs to provide metadata without listing all provider models first.
  • Applies explicit metadata consistently across getModelMetadata(), hasModelMetadata(), and listed model metadata for matching listed IDs.
  • Keeps explicit metadata request-local and memoized per directory instance instead of storing synthetic metadata in the shared models cache.

Fixes #230.

Why

Explicit provider/model selection currently calls the provider model-list endpoint before the model can be instantiated. For providers that accept explicit/current model IDs, that means getProviderModel( 'openai', 'gpt-5.4' ) can fail on a transient GET /models issue before the actual generation endpoint has a chance to validate the explicitly requested model.

Testing

  • composer test -- --filter AbstractApiBasedModelMetadataDirectoryTest
  • composer lint

AI assistance

  • AI assistance: Yes
  • Tool(s): OpenCode (GPT-5.6 Sol)
  • Used for: Drafting the patch and tests, including the batched explicit-metadata follow-up; Chris reviewed the behavior on a local Studio site and remains responsible for the change.

@github-actions

github-actions Bot commented May 2, 2026

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: chubes4 <extrachill@git.wordpress.org>
Co-authored-by: JasonTheAdams <jason_the_adams@git.wordpress.org>
Co-authored-by: felixarntz <flixos90@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@codecov

codecov Bot commented May 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.60%. Comparing base (6317042) to head (7c6b04b).
⚠️ Report is 38 commits behind head on trunk.

Additional details and impacted files
@@             Coverage Diff              @@
##              trunk     #231      +/-   ##
============================================
- Coverage     88.12%   86.60%   -1.53%     
- Complexity     1213     1335     +122     
============================================
  Files            60       68       +8     
  Lines          3934     4314     +380     
============================================
+ Hits           3467     3736     +269     
- Misses          467      578     +111     
Flag Coverage Δ
unit 86.60% <100.00%> (-1.53%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The OpenAI-compatible abstraction is about HTTP/JSON shape, not OpenAI's
model namespace. Drop the gpt-/o3/dall-e prefix gating and the synthetic
text-generation metadata override; the per-provider override belongs in
ai-provider-for-openai (and similar repos for other compatible providers).

The generic createModelMetadataForExplicitModelId() hook on
AbstractApiBasedModelMetadataDirectory is unchanged so providers can still
opt in.

@felixarntz felixarntz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chubes4 While this addresses the reported concern, it creates data inconsistencies when considering the API surface of this class holistically.

I think there's a path forward, but it requires some modifications in the approach.

Comment thread src/Providers/ApiBasedImplementation/AbstractApiBasedModelMetadataDirectory.php Outdated
Comment thread src/Providers/ApiBasedImplementation/AbstractApiBasedModelMetadataDirectory.php Outdated
@chubes4

chubes4 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

Updated this to address the consistency concern:

  • getModelMetadata() and hasModelMetadata() now check explicit metadata before the list/cache path, so provider-local metadata is authoritative when available.
  • listModelMetadata() still performs the required list request, then overlays explicit metadata for matching listed model IDs so listed data and direct lookup agree for the same ID.
  • Explicit metadata is memoized only for the directory instance, avoiding persistence/cache invalidation concerns for synthetic metadata.

Verified with:

  • composer test -- --filter AbstractApiBasedModelMetadataDirectoryTest
  • composer lint

@felixarntz felixarntz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chubes4 I think this looks good, though one optimization recommendation remains.

Comment on lines +128 to +130
if (!array_key_exists($modelId, $this->explicitModelMetadataCache)) {
$this->explicitModelMetadataCache[$modelId] = $this->createModelMetadataForExplicitModelId($modelId);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of calling createModelMetadataForExplicitModelId for every single model ID (including all the ones that inevitably won't have explicit model metadata), why don't we instead make $this->explicitModelMetadataCache a null value initially that gets populated upon the first call for a model, but with all relevant models that have explicit metadata?

since explicit model metadata will have to be provided in "manual" code, I doubt you'd have more than a dozen or so entries there. But there may be 100s of model IDs in a provider, and many providers probably wouldn't even implement explicit model ID metadata at all. So I think it would be worth optimizing performance for the basic case that you get out of the box (no explicit model metadata).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 7c6b04b. The core hook now receives all unchecked candidate IDs in one call and returns a keyed override map. The directory memoizes both returned metadata and omitted IDs, so providers using the default implementation incur one O(1) callback for an entire listed set rather than one callback per model. The dependent OpenAI implementation remains dynamic by matching future model IDs within the supplied batch: WordPress/ai-provider-for-openai#23

Verified with the focused 11-test directory suite, full lint, and an integrated OpenAI gpt-5.4 lookup that completed without a models request.

AI assistance: OpenCode (GPT-5.6 Sol) implemented and verified the batch follow-up; Chris reviewed and remains responsible for the change.

AI assistance: OpenCode (GPT-5.6 Sol) implemented the batch hook and tests; Chris reviewed and remains responsible for the change.
@chubes4

chubes4 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Thank you, @felixarntz!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explicit model instantiation should not require live model-list metadata fetch

3 participants