Skip to content

feat: gpt caching - #80

Open
schiwekM wants to merge 12 commits into
mainfrom
feat/gpt-caching
Open

feat: gpt caching#80
schiwekM wants to merge 12 commits into
mainfrom
feat/gpt-caching

Conversation

@schiwekM

@schiwekM schiwekM commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Prompt caching is now also applied to GPT models.

Have you...

  • Added relevant entry to the change log?

@schiwekM
schiwekM requested review from a team as code owners September 4, 2026 13:27
@schiwekM
schiwekM requested a review from Akatuoro September 4, 2026 13:34
@Akatuoro

Copy link
Copy Markdown
Contributor

AI review:

lib/models/aicore.js
  ● 174 [correctness] Tool `cache_control` injection for Claude is removed with no replacement, silently disabling tool-definition caching.
  ● 126 [correctness] `mergeOrchestrationConfig` accesses `config.promptTemplating.model` without null-checking `config.promptTemplating`, crashing if super returns a config without that field.
lib/models/aicore-caching.js
  ●  36 [correctness] `withPromptCachingOptions` replaces multi-breakpoint per-message injection with a single top-level `cache_control`, reducing Claude caching coverage to one message instead of up to four.
  ● 100 [correctness] `_cacheKeyPart` truncates to 32 chars, causing `prompt_cache_key` collisions between Claude 3.5 Sonnet versions deployed simultaneously in AI Core.
  ●  59 [correctness] `buildPromptCacheKey` uses `user || thread`, so a `null` or empty-string `userId` silently falls through to `thread_id`, producing the wrong affinity bucket.
  ●  88 [efficiency]  `isGptExtendedRetentionModel` and `isGpt56OrLater` are both called without short-circuit in `isPromptCachingModel`, computing `_gptVersion` twice per call.
lib/telemetry/chat-tracing.js
  ● 169 [correctness] `_detectCacheRequest` checks `modelOptions.params.prompt_cache_key` but the cache key is never stored in model params at construction time, making that branch permanently dead.

The selection logic seems quite complicated, as evidenced in the bullet points above... can we boil it down to easier criteria?
Also, aicore-caching.js should not be located in lib/models, that is reserved for the impls.

Comment thread lib/models/aicore-caching.js Outdated
@Akatuoro

Copy link
Copy Markdown
Contributor

Works for anthropic provider. For aicore provider, it works with gpt-5.5 and anthropic models. For other gpt models (gpt-5.2, gpt-5.6-sol, ...), I get Error while iterating over SSE stream.

Also for aicore, I see these in the log:

[agents] - [mlflow] GET /api/2.0/mlflow/registered-models/get?name=srv%2Ftravel-agent%2FAGENTS.md → 400: {"error_code": "INVALID_PARAMETER_VALUE", "message": "Invalid model name 'srv/travel-agent/AGENTS.md'. Names cannot contain '/' or ':'.", "sqlstate": "KAM00", "error_class": "INVALID_PARAMETER_VALUE"}
[agents] - [mlflow] POST /api/2.0/mlflow/registered-models/create → 400: {"error_code": "INVALID_PARAMETER_VALUE", "message": "Invalid model name 'srv/travel-agent/AGENTS.md'. Names cannot contain '/' or ':'.", "sqlstate": "KAM00", "error_class": "INVALID_PARAMETER_VALUE"}
[agents] - [mlflow] POST /api/2.0/mlflow/model-versions/create → 400: {"error_code": "INVALID_PARAMETER_VALUE", "message": "Invalid model name 'srv/travel-agent/AGENTS.md'. Names cannot contain '/' or ':'.", "sqlstate": "KAM00", "error_class": "INVALID_PARAMETER_VALUE"}

@schiwekM schiwekM mentioned this pull request Sep 12, 2026
1 task
@schiwekM

Copy link
Copy Markdown
Collaborator Author

The mlflow errors are fixed in the other PR. gpt 5.2 is not available with version latest in AI core causing the error and got-5.6-sol did not accept temperature = 0.
For the first scenario I am adjusting the AI Core class that the errors are caught and correctly thrown avoiding the generic error message.
For 5.6-sol I am adding a condition that models we know don't accept a temperature of zero do not get it.

Comment thread lib/models/aicore.js Outdated
options.params || (deepAgent ? { max_tokens: 4096, temperature: 0 } : cds.env.agents?.params)
const params = withPromptCachingParams(model, rawParams)
if (params.temperature === 0 && noTemp0Support(model)) {
params.temperature = 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting temperature to 1 ourselves is a bad idea. We should ideally not have to provide it at all

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. Deleting it should work as well and I switched to that now.

@schiwekM
schiwekM requested a review from Akatuoro September 13, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants