Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
43ec04d
feat: scope evaluators to the current project on create
quinn-galileo Oct 14, 2026
bd613de
fix: default project_ids on all evaluators and share the no-current-p…
quinn-galileo Oct 14, 2026
c5db4da
fix: cover fetched evaluators and tidy scope wording
quinn-galileo Oct 14, 2026
bd119d5
chore: tidy comments
quinn-galileo Oct 14, 2026
96f1114
chore: merge main into scope-on-create branch
quinn-galileo Oct 19, 2026
49bef14
fix: remove committed conflict markers and tighten scope validation
quinn-galileo Oct 19, 2026
9fcea53
fix: keep requested scope private and clarify scope errors
quinn-galileo Oct 19, 2026
9e7b850
chore: dedupe the merged ScorerName enum in the OpenAPI import
quinn-galileo Oct 20, 2026
535f5e0
chore: regenerate the API client from the current Client API spec
quinn-galileo Oct 20, 2026
066f32d
fix: adapt hand-written code to the regenerated client
quinn-galileo Oct 20, 2026
2221540
test: parse a project whose permissions include configure_rate_limit
quinn-galileo Oct 20, 2026
de519d0
chore: changelog for the regenerated API client
quinn-galileo Oct 20, 2026
f114686
test: cover the ProjectCreate body sent by Projects.create
quinn-galileo Oct 20, 2026
9c7a5c6
chore: share the project response fixture and mark the client changes…
quinn-galileo Oct 20, 2026
3782fdb
chore: tighten the changelog for the regenerated client
quinn-galileo Oct 20, 2026
42eb8e5
chore: soften the removed-enum note
quinn-galileo Oct 20, 2026
5fb17e7
chore: tidy comments
quinn-galileo Oct 20, 2026
99eb31b
chore: keep enum values older servers still return in the OpenAPI import
quinn-galileo Oct 20, 2026
e5dac8a
chore: regenerate the API client with the restored enum values
quinn-galileo Oct 20, 2026
4b266c0
test: parse a legacy task_type and state test intent
quinn-galileo Oct 20, 2026
2bfc0f5
chore: complete the changelog for the regenerated client
quinn-galileo Oct 20, 2026
42aa4fd
chore: check every patched enum exists before the OpenAPI import patches
quinn-galileo Oct 20, 2026
454c0dd
chore: include StepType in the enum check and correct the ground_trut…
quinn-galileo Oct 20, 2026
a698e55
chore: rewrap a changelog line
quinn-galileo Oct 20, 2026
f22cebe
chore: merge client regeneration into scope-on-create branch
quinn-galileo Oct 20, 2026
15f00ce
chore: fix duplicate Changed heading in the changelog
quinn-galileo Oct 20, 2026
29b5453
chore: correct the scope validation wording in the changelog
quinn-galileo Oct 20, 2026
a989ff4
Merge remote-tracking branch 'origin/main' into chore/sao-9342-t0-reg…
quinn-galileo Oct 30, 2026
56b7908
chore: regenerate the API client from the current server spec
quinn-galileo Oct 30, 2026
3c299e0
chore: note that keyword-built scorer configs turn ground truth off
quinn-galileo Oct 30, 2026
e531d69
chore: require splunk-ao 0.7.0 for project-scoped evaluator creates
quinn-galileo Oct 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
62 changes: 62 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,68 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- `LlmEvaluator`, `CodeEvaluator`, `DecisionEvaluator`, `Evaluators.create_custom_llm_evaluator()`,
`Evaluators.create_decision_evaluator()` and the module-level `create_custom_llm_evaluator()` and
`create_decision_evaluator()` accept `project_ids` and `is_global` to choose the scope of a new evaluator.
Passing `is_global=True` together with `project_ids`, an empty `project_ids` without `is_global=True`, or
`is_global=False` with no `project_ids` and no current project raises `ValidationError` before any request is
sent.
- `Evaluator.is_global` and `Evaluator.scope_projects` report the scope the API returned, after
`create()`, `get()`, `list()` and `refresh()`.

### Changed

- New evaluators are scoped to the current project by default: the decorator context project, then
`SPLUNK_AO_PROJECT_ID`, then `SPLUNK_AO_PROJECT`. If that project cannot be found, creation raises
`NotFoundError` rather than falling back to a global evaluator. When no current project is set and no scope is
passed, no scope is sent and the server default (global, org admins only) applies. Org admins who relied on creating
global evaluators with a project set must now pass `is_global=True`.
The current project must already exist: unlike logging, creating an evaluator does not create it. Create it first,
or pass `project_ids` or `is_global=True`.
- Creating project-scoped evaluators requires `splunk-ao` 0.7.0 or later. Earlier versions send no scope, so on servers
with metric RBAC enabled, non-admin users get a 403 when creating an evaluator.
- **Breaking:** the low-level API client (`splunk_ao.resources`) is regenerated from the current Client API
spec. Code that imports generated symbols directly may need updating:
- `ScorerConfig` is now `RuntimeScorerConfig`. It keeps the old fields, adds new ones and makes `scorer_type`
optional. The field order changed, so a positional `ScorerConfig(id, scorer_type)` call must become
`RuntimeScorerConfig(id=..., scorer_type=...)`. Keyword construction now also sends the defaults
`ground_truth=False`, `threshold=0.5` and `requires_tools_in_llm_span=False`; building one from a scorer
with `from_dict` takes `ground_truth` from the scorer and leaves the other two unset. The server keeps an
explicit `ground_truth`, so a keyword-built config for a ground-truth scorer turns ground truth off; build it
from the scorer with `from_dict` or pass `ground_truth=True`.
`Experiments.create(scorers=...)`, `update_scorer_settings()` and `create_metric_configs()` take or return
`RuntimeScorerConfig`.
- `CoreScorerName` is renamed to `ScorerName`. The old `ScorerName`, whose members were internal
underscore-prefixed values (for example `ScorerName.VALUE_1 = "_context_adherence_luna"`), is gone; use the
public names such as `ScorerName.CONTEXT_ADHERENCE`.
- Removed endpoints: experiment metrics (`/projects/{project_id}/experiments/metrics`), experiment metric settings
(`/projects/{project_id}/experiments/{experiment_id}/metric_settings`) and the Vegas gateway integration
(`/integrations/vegas_gateway`).
- Removed models: the `Customized*GPTScorer` and `*Template` scorer families, `ChunkAttributionUtilizationScorer`,
`FineTunedScorer`, `RegisteredScorer`, `ScorerType`, `ScorersConfiguration`, `MetricCritiqueColumnar`,
`MetricCritiqueContent`, `ExtendedSessionRecordWithChildren` (replaced by
`ExtendedReadSessionRecordWithChildren`), `VegasGatewayIntegration` and `VegasGatewayIntegrationCreate`.
- Removed fields:
- `BaseScorerVersionResponse.input_type` (returned by `create_custom_llm_evaluator()` and the other evaluator
create functions), and `input_type`, `output_type` and `scoreable_node_types` on `BaseScorerVersionDB`.
- `BaseScorer.sub_scorers` and `ProjectCreate.create_example_templates`.
- `scorer_type` on `MetricComputing`, `MetricError`, `MetricFailed`, `MetricNotApplicable`, `MetricNotComputed`,
`MetricPending`, `MetricRollUp` and `MetricSuccess`, and `critique` on `MetricRollUp` and `MetricSuccess`.
- `prompt_scorers_configuration`, `prompt_customized_scorers_configuration`,
`prompt_finetuned_scorers_configuration`, `prompt_registered_scorers_configuration`, `sub_scorers`,
`store_metric_ids`, `stream_metrics` and `xray` on `CreateJobRequest` and `CreateJobResponse`.
- Removed enum members: `ProjectAction.UPDATE`, `ProjectType.LLM_MONITOR` and `ProjectType.PROMPT_EVALUATION`.

### Fixed

- The API client now knows the permission and enum values that current servers return, so reading projects and
other resources no longer fails on them. For example, `Projects().get(id=...)` raised
`ValueError: 'configure_rate_limit' is not a valid ControlResourceAction` (the project permission enum
`ProjectAction` lacked `configure_rate_limit`). Values that older servers still return, `vegas_gateway` for
integrations and task types 7, 9 and 12, are kept so reads from those servers keep working.

## [0.6.1]

### Fixed
Expand Down
Loading
Loading