Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ Use this file when changing this repository. It is not an operating guide for us
## Architecture and boundaries

- `dataiku_mcp/server.py` constructs the FastMCP server, request middleware, and stdio/HTTP runners. `dataiku_mcp/__init__.py` re-exports that public server API and imports each tool module so decorators register their tools.
- `dataiku_mcp/tools/` contains thin adapters over the synchronous `dataikuapi` client. Shared client authentication and executors live at the package root; validation, serialization, and error helpers live under `dataiku_mcp/tools/utils/`.
- `dataiku_mcp/tools/` contains thin adapters over the synchronous `dataikuapi` client. Shared client authentication and executors live at the package root; shared validation, serialization, and redaction live under `dataiku_mcp/utils/`; tool-specific helpers remain under `dataiku_mcp/tools/utils/`.
- `dataiku_mcp/operations/` owns operation models and the explicit catalog; `operations/handlers/` contains undecorated SDK adapters. The four MCP entry points remain in `dataiku_mcp/tools/operations.py`.
- `dataiku_mcp/tools/cobuild.py` owns retained Cobuild conversations. Cobuild is the default path for constructing or modifying project-level flow and analytic assets.
- Cobuild owns construction and modification of project-level flow and analytic assets. Direct writes are allowed only for the fixed exception categories defined by the **Cobuild Write-Routing Convention** in `CODING_STANDARDS_AND_STRUCTURE.md`; do not infer permission for a new direct write from existing implementation.
- `skills/dataiku-headless/SKILL.md` is the operator-facing router. Its `references/` directory owns object-specific inspection, mutation, and verification workflows.
Expand All @@ -37,7 +38,7 @@ Do not copy volatile inventories, versions, or command details into this file wh
- Keep tool handlers `async def`. Put every blocking SDK or filesystem operation inside `run_blocking(...)`; use `run_cobuild_blocking(...)` only for retained Cobuild turns.
- Capture any client or instance identity needed for consistency before the first `await` when a concurrent instance switch could otherwise change the target.
- Keep handlers thin: validate boundary invariants, call the SDK, shape a compact response, and let Dataiku validate deeper domain rules.
- Reuse helpers from `tools/utils/` instead of adding local variants. Use `compact_json`, `columnar`, and `omit_empty` for consistent responses where appropriate.
- Reuse shared helpers from `utils/` and tool-specific helpers from `tools/utils/` instead of adding local variants. Use `compact_json`, `columnar`, and `omit_empty` for consistent responses where appropriate.
- Raise clear exceptions for whole-operation failures. Return structured per-item or uncertain-outcome payloads only when callers need partial results or durable job/run identities for recovery.
- Never return API keys, credentials, unredacted connection secrets, or sensitive exception text.
- Represent destructive intent with semantic parameters such as `overwrite`, `drop_data`, or a replace/run mode. Do not add `confirm*` parameters; confirmation is a harness concern.
Expand Down
14 changes: 9 additions & 5 deletions CODING_STANDARDS_AND_STRUCTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ export DKU_API_KEY="your-api-key"
| What to change | Where |
| --- | --- |
| MCP tool logic | `dataiku_mcp/tools/**/*.py` |
| Shared validation helpers | `dataiku_mcp/tools/utils/validation.py` |
| Shared validation helpers | `dataiku_mcp/utils/validation.py` |
| Workflow prompts | `dataiku_mcp/prompts/workflows.py` |
| Project/dataset/folder/recipe/ML skills | `skills/**/SKILL.md` |
| Cobuild conversation tools | `dataiku_mcp/tools/cobuild.py` |
Expand Down Expand Up @@ -74,7 +74,7 @@ export DKU_API_KEY="your-api-key"
## Tool Input Validation
- Assume FastMCP/Pydantic type validation for tool signatures; avoid duplicating type checks in tool code unless there is a clear compatibility/safety reason.
- Keep validation for structured inputs (JSON payloads/lists), enum-like parameters, and semantic risk parameters (`overwrite`, `drop_data`, replace modes, recursive build modes).
- Put reusable validation logic in `dataiku_mcp/tools/utils/validation.py` and reuse those helpers from tool modules instead of creating ad-hoc local validators.
- Put reusable validation logic in `dataiku_mcp/utils/validation.py` and reuse those helpers from tool modules instead of creating ad-hoc local validators.
- When reducing validation, preserve existing destructive-action safeguards and return clear errors from downstream API calls.

## Async And Blocking Style
Expand Down Expand Up @@ -110,9 +110,13 @@ export DKU_API_KEY="your-api-key"
## Fixed Tool Surface

- `runtime/run_mcp.py` is the single launcher. It requires `--transport stdio` for the local plugin or `--transport http` for the authenticated Streamable HTTP deployment. Keep the transports explicitly selected and preserve the same registered tool catalog.
- Register one directly visible tool catalog. Do not add an MCP search mode.
- Classify explicit tools with `PROJECT_READ_POLICY`, `PROJECT_EXECUTION_POLICY`, `ADMIN_POLICY` or `PROJECT_EDIT_POLICY`; use `INSTANCE_MANAGEMENT_POLICY` for connection controls. Execution includes status/polling, administration includes instance inspection and project folders, and project edits include project Cobuild. Only connection management is exempt from requiring a selected profile.
- `SUPPORTED_DSS_INSTANCE_TYPES` defines the currently supported backend types independently of all valid `InstanceType` values. Agent Management, Govern and Deployer remain configurable profiles with instance controls only until their product support lands.
- Register one stable, directly visible MCP tool catalog. Do not change registration when instances switch. The bounded operation-registry pilot exposes fixed search/describe/read/write tools; operation discovery does not dynamically register MCP tools.
- Classify v1 tools with `PROJECT_READ_POLICY`, `PROJECT_EXECUTION_POLICY`, `ADMIN_POLICY` or `PROJECT_EDIT_POLICY`; use `API_OPERATIONS_POLICY` for shared operation tools and `INSTANCE_MANAGEMENT_POLICY` for connection controls. The categories describe purpose: execution includes status/polling, administration includes instance inspection and project folders, and project edits include project Cobuild.
- Only the `INSTANCE_MANAGEMENT_POLICY` group is exempt from requiring a selected profile. All v1 and v2 Dataiku tools, including operation search/describe, require one. Read/write execution handlers validate their selected operation directly.
- Every tool declares allowed_instance_types metadata. Middleware enforces it against the pinned request instance before backend token exchange. Operation handlers additionally enforce the selected operation’s instance policy, read/write effect, arguments and transport before invoking its SDK adapter; missing declarations fail closed.
- `SUPPORTED_DSS_INSTANCE_TYPES` identifies types served by the DSS client independently of project-tool policies.
- Product operations use public Python SDK methods only; do not call private transport helpers or add direct endpoint fallbacks. Deployer supports stdio and HTTP; other product support is deferred.
- Operation arguments use one Pydantic model for discovery and execution. Never accept caller-supplied HTTP methods or paths. Keep backend requirements and response projections explicit; catalog presence does not verify live server support. Preserve Cobuild routing for in-project asset construction.
- The registered set is the contract; `tests/test_tool_surface.py` pins the exact catalog. Any tool add/remove/rename updates that pinned set in the same change.

## Guardrails
Expand Down
31 changes: 24 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,20 @@ Dataiku Headless is an MCP server and agent skill library for operating Dataiku

Cobuild runs as a retained conversation through MCP tools. This repository intentionally keeps its own tool surface small: inspection tools, three deterministic executions of existing assets (`build_datasets`, `run_recipe`, and `run_scenario`), narrow configuration actions such as selecting which container an existing object runs in, and a few bootstrap actions that Cobuild cannot perform, such as creating a project or uploading a local file.

Project and API Deployer operations discover infrastructures, published versions and deployments,
create deployment configurations, change target versions, apply updates asynchronously,
and inspect health, governance codes and update history. They operate on already published
bundles and API services. See the [Deployer guide](skills/dataiku-headless/references/deployer.md)
for node selection, production effects and coverage limits.

Direct project reads, execution and administration support Design, Automation.
Project edits and Cobuild require Design. The four shared operation tools expose
13 public SDK actions for Deployer, restricted to their matching profiles.
Search returns all matching operation summaries; describe loads individual schemas
on demand, and separate read/write tools execute them.
See the [operation guide](skills/dataiku-headless/references/operations.md).


## Capability reference

For a quick reference to what Headless can inspect, what Cobuild builds, and the
Expand All @@ -133,7 +147,7 @@ To build or update a Dataiku plugin, follow the

## MCP Server

`dataiku_mcp` is a FastMCP server that exposes Dataiku operations as typed, async MCP tools. Tools are organized by domain: projects, project folders, flow, connections, datasets, data quality, managed folders, recipes, machine learning, insights, dashboards, scenarios, WebApps, wikis, agents, LLMs and knowledge banks, instance plugins, job management, administrative tasks, and Cobuild conversations.
`dataiku_mcp` is a FastMCP server that exposes Dataiku operations as typed, async MCP tools. Tools are organized by domain: projects, project folders, flow, connections, datasets, data quality, managed folders, recipes, machine learning, insights, dashboards, scenarios, WebApps, wikis, agents, LLMs and knowledge banks, instance plugins, Project and API Deployer, job management, administrative tasks, and Cobuild conversations.

- Async execution for all Dataiku API calls
- Progress notifications for long-running operations
Expand Down Expand Up @@ -180,11 +194,8 @@ Each profile requires `instance_type`: `design`, `automation`, `deployer`, `gove
`agent-management`. Existing stdio profiles missing this field are automatically
assigned `design` and rewritten on disk before loading. This temporary migration
is scheduled for deprecation by 0.9.0; existing values are preserved and validated.
The configured type determines which tools can run; see the
[instance-type policy](docs/capabilities.md#instance-types). Agent Management, Govern and Deployer
currently support instance controls only. Their status is `unsupported`, without
a DSS connection check. The setup page allows these profiles to be saved without testing.
Design and Automation require a successful connection test.
The declared type selects tool policy; backend features and permissions are validated by Dataiku. A local Deployer on a Design or Automation node requires a separate `deployer` profile for that URL.
Design, Automation and Deployer use the DSS client. Agent Management and Govern remain configurable profiles with instance controls only and report unsupported without a backend call.

**Environment override.** To select an explicit target, including inside a Code
Studio, set these three variables in your environment or copy
Expand Down Expand Up @@ -246,6 +257,11 @@ The Dataiku Headless marketplace plugin uses stdio transport. For customer-manag
├── dataiku_mcp/
│ ├── auth.py # Dataiku client creation and HTTP token exchange
│ ├── executors.py # Shared blocking and Cobuild executors
│ ├── operations/ # Operation models, catalog, and undecorated handlers
│ │ ├── models.py # Typed operation definitions and argument schemas
│ │ ├── catalog.py # Explicit registration and lookup
│ │ └── handlers/deployer.py # Internal Deployer SDK adapters
│ ├── utils/ # Shared validation, serialization, and redaction
│ ├── tools/
│ │ ├── agents.py # Agent/agent-version/agent-tool inspection tools
│ │ ├── agent_reviews.py # Agent review/test/run inspection tools
Expand All @@ -258,6 +274,7 @@ The Dataiku Headless marketplace plugin uses stdio transport. For customer-manag
│ │ ├── data_quality.py # Dataset Data Quality rule inspection tools
│ │ ├── dashboards.py # Dashboard inspection tools
│ │ ├── datasets.py # Dataset inspection tools + local-file upload writes
│ │ ├── operations.py # Shared discovery/read/write MCP tools
│ │ ├── evaluation_stores.py # Evaluation Store inspection tools
│ │ ├── flow.py # Flow inspection tools
│ │ ├── instances.py # Multi-instance switching tools
Expand All @@ -277,7 +294,7 @@ The Dataiku Headless marketplace plugin uses stdio transport. For customer-manag
│ │ ├── project_libraries.py # Project library inspection/search + local-file write
│ │ ├── recipes.py # Recipe inspection tools
│ │ ├── machine_learning/ # ML analysis/saved-model inspection tools
│ │ └── utils/ # Tool validation and response-shaping utilities
│ │ └── utils/ # Tool-specific helpers
│ ├── config/ # Models, stdio/HTTP configuration, and request routing
│ ├── server.py # FastMCP construction, middleware, and transport startup
│ ├── setup_server.py # Temporary loopback page used by URL elicitation
Expand Down
1 change: 1 addition & 0 deletions dataiku_mcp/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
licensing,
llms_and_knowledge_banks,
managed_folders,
operations,
plugins,
project_folders,
project_libraries,
Expand Down
13 changes: 6 additions & 7 deletions dataiku_mcp/instance_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,16 @@

from .config.models import InstanceType

# Product-node support is added separately from this Design/Automation foundation.
SUPPORTED_DSS_INSTANCE_TYPES = frozenset({"design", "automation"})
# Instance types currently supported through DSSClient.
SUPPORTED_DSS_INSTANCE_TYPES = frozenset({"design", "automation", "deployer"})

# These are @mcp.tool metadata policies. V1 tools expose individual actions;
# project authoring still routes through Cobuild and its fixed direct exceptions.
PROJECT_READ_POLICY = {"allowed_instance_types": sorted(SUPPORTED_DSS_INSTANCE_TYPES)}
PROJECT_EXECUTION_POLICY = {
"allowed_instance_types": sorted(SUPPORTED_DSS_INSTANCE_TYPES)
}
ADMIN_POLICY = {"allowed_instance_types": sorted(SUPPORTED_DSS_INSTANCE_TYPES)}
PROJECT_READ_POLICY = {"allowed_instance_types": ["design", "automation"]}
PROJECT_EXECUTION_POLICY = {"allowed_instance_types": ["design", "automation"]}
ADMIN_POLICY = {"allowed_instance_types": ["design", "automation"]}
PROJECT_EDIT_POLICY = {"allowed_instance_types": ["design"]}
API_OPERATIONS_POLICY = {"allowed_instance_types": ["deployer"]}
INSTANCE_MANAGEMENT_POLICY = {"allowed_instance_types": list(get_args(InstanceType))}


Expand Down
15 changes: 15 additions & 0 deletions dataiku_mcp/operations/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Copyright 2026 Dataiku SAS
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Operation definitions and catalog."""
Loading
Loading