Avoid listing models for explicit text model IDs - #231
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## trunk #231 +/- ##
============================================
- Coverage 88.12% 86.60% -1.53%
- Complexity 1213 1335 +122
============================================
Files 60 68 +8
Lines 3934 4314 +380
============================================
+ Hits 3467 3736 +269
- Misses 467 578 +111
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4b2547d to
3e79f62
Compare
The OpenAI-compatible abstraction is about HTTP/JSON shape, not OpenAI's model namespace. Drop the gpt-/o3/dall-e prefix gating and the synthetic text-generation metadata override; the per-provider override belongs in ai-provider-for-openai (and similar repos for other compatible providers). The generic createModelMetadataForExplicitModelId() hook on AbstractApiBasedModelMetadataDirectory is unchanged so providers can still opt in.
felixarntz
left a comment
There was a problem hiding this comment.
@chubes4 While this addresses the reported concern, it creates data inconsistencies when considering the API surface of this class holistically.
I think there's a path forward, but it requires some modifications in the approach.
|
Updated this to address the consistency concern:
Verified with:
|
felixarntz
left a comment
There was a problem hiding this comment.
@chubes4 I think this looks good, though one optimization recommendation remains.
| if (!array_key_exists($modelId, $this->explicitModelMetadataCache)) { | ||
| $this->explicitModelMetadataCache[$modelId] = $this->createModelMetadataForExplicitModelId($modelId); | ||
| } |
There was a problem hiding this comment.
instead of calling createModelMetadataForExplicitModelId for every single model ID (including all the ones that inevitably won't have explicit model metadata), why don't we instead make $this->explicitModelMetadataCache a null value initially that gets populated upon the first call for a model, but with all relevant models that have explicit metadata?
since explicit model metadata will have to be provided in "manual" code, I doubt you'd have more than a dozen or so entries there. But there may be 100s of model IDs in a provider, and many providers probably wouldn't even implement explicit model ID metadata at all. So I think it would be worth optimizing performance for the basic case that you get out of the box (no explicit model metadata).
There was a problem hiding this comment.
Updated in 7c6b04b. The core hook now receives all unchecked candidate IDs in one call and returns a keyed override map. The directory memoizes both returned metadata and omitted IDs, so providers using the default implementation incur one O(1) callback for an entire listed set rather than one callback per model. The dependent OpenAI implementation remains dynamic by matching future model IDs within the supplied batch: WordPress/ai-provider-for-openai#23
Verified with the focused 11-test directory suite, full lint, and an integrated OpenAI gpt-5.4 lookup that completed without a models request.
AI assistance: OpenCode (GPT-5.6 Sol) implemented and verified the batch follow-up; Chris reviewed and remains responsible for the change.
AI assistance: OpenCode (GPT-5.6 Sol) implemented the batch hook and tests; Chris reviewed and remains responsible for the change.
|
Thank you, @felixarntz! |
Summary
getModelMetadata(),hasModelMetadata(), and listed model metadata for matching listed IDs.Fixes #230.
Why
Explicit provider/model selection currently calls the provider model-list endpoint before the model can be instantiated. For providers that accept explicit/current model IDs, that means
getProviderModel( 'openai', 'gpt-5.4' )can fail on a transientGET /modelsissue before the actual generation endpoint has a chance to validate the explicitly requested model.Testing
composer test -- --filter AbstractApiBasedModelMetadataDirectoryTestcomposer lintAI assistance