Add redshift-support-specialist skill and custom agent - #30
Open
andrehass wants to merge 35 commits into
Open
Conversation
Adds the redshift-support-specialist skill for query optimization, operational reviews, disaster recovery guidance, incident detection guidance, and cost optimization on Amazon Redshift provisioned clusters and Serverless workgroups. The skill talks to Redshift exclusively through the six tools exposed by the awslabs.redshift- mcp-server MCP server (list_clusters, list_databases, list_schemas, list_tables, list_columns, execute_query) and includes a ready-to-use serverless (Lambda + API Gateway) deployment of that MCP server. Also adds a companion redshift-support-specialist custom agent that orchestrates the skill and forces active-chat-only execution (never background), and documents both the deployment and the skill/agent usage in their respective READMEs. Updates llms.txt with the new skill entry.
Removes the Lambda Function URL endpoint from the Redshift MCP server deployment, leaving a single API Gateway REST API (execute-api, AWS_IAM/SigV4) as the only way to reach the MCP server. - template.yaml: drops FunctionUrlConfig, CallerInvokePermission, and the Function URL invoke policy/output. - deploy.sh: replaces Function URL creation with API Gateway provisioning (REST API, /mcp resource, method, integration with caller-identity passthrough, Prod stage deployment); caller grants now use put-role-policy for execute-api:Invoke + lambda:InvokeFunction. - deployer-permissions-policy.json: swaps Function URL Lambda actions for apigateway:* management and iam:PutRolePolicy. - scripts/*.py: default MCP_SIGV4_SERVICE to execute-api and update usage examples to the API Gateway URL shape. - README.md / sam-app/README.md: updated architecture, deploy, auth, invoke-grant, testing, and teardown sections to match the single-endpoint setup.
Adds a "Quick deploy (no prompts)" command as the primary deploy instruction in both the main skill README and deployment/sam-app/README.md: a single sam deploy call using --capabilities CAPABILITY_NAMED_IAM, --resolve-s3, --no-confirm-changeset, and --no-fail-on-empty-changeset, with all template defaults and --parameter-overrides for customization. Always using CAPABILITY_NAMED_IAM (a superset of CAPABILITY_IAM) removes the need to branch on whether CreateDevOpsAgentRole is set, so the same command works either way and avoids the "Requires capabilities: [CAPABILITY_NAMED_IAM]" changeset failure users hit with sam deploy --guided's default capability. The previous sam deploy --guided flow is kept as a secondary "Guided deploy" option for anyone who wants interactive prompts, with the capability gotcha still called out there. Verified with sam validate --lint against the current template.
Restructures the main skill README's Prerequisites section into a strict, numbered dependency chain (Step 1: Agent Space, Step 2: deploy the MCP server + test it, Step 3: register/connect the MCP server to the Agent Space) instead of listing the capability-provider registration steps before the deployment steps they depend on. Adds a "Setup Overview" block right after the architecture diagram with anchor links to each step, so a first-time reader sees the full flow before the details. Adds a real "Optional: Create a Custom Agent" section linking to custom-agents/redshift-support-specialist/, which was previously only mentioned once in a buried Skill Contents note. Custom agent README and deployment/README.md cross-references updated to the new step anchors.
Adds the two-step GitHub connection setup (register at Capability Providers, then attach via the Agent Space's Capabilities tab -> Pipeline section) to the skill's "Option A: Import from GitHub" instructions, based on the actual console navigation path (Capability Providers is account-level and separate from the Agent Space's own Capabilities tab).
Renumbers the setup flow to align Setup Overview with the actual step headings: Step 1 (MCP Server Deployment), Step 2 (Connect the MCP server to your Agent Space), Step 3 (Create the redshift-support-specialist skill), Step 4 (Create the Custom Agent, no longer marked optional), Step 5 (How to Use the Skill). Consolidates all setup prerequisites (Agent Space, deploy tools, IAM permissions to deploy) into a single Prerequisites section before Step 1, with detail pointers to deployment/README.md. Moves Limitations, Agent Types, Architecture, and How the Pieces Fit Together to the end of the document, after Skill Contents and before License, so the step-by-step setup flow reads uninterrupted. Updates cross-references in custom-agents/redshift-support-specialist/README.md and deployment/README.md to match the renumbered step anchors.
The Lambda's IAM execution role had no explicit name in the SAM template, so CloudFormation auto-generated one with a random suffix (e.g. redshift-mcp-RedshiftMcpFunctionRole-LWxE1aGLRtLI) -- leaving no stable name to grant database-level permissions against. Earlier docs also referenced the Lambda *function* name instead of its *execution role*, and used the IAM: prefix (for IAM users) instead of IAMR: (for IAM roles) when deriving the Redshift database username. - sam-app/template.yaml: adds an explicitly named RedshiftMcpFunctionRole (redshift-mcp-lambda-execution-role) and points the function's Role property at it. Adds GrantSysMonitorCommand and GrantTableInfoCommand stack outputs that generate the exact GRANT SQL with the real deployed role name. - deployer-permissions-policy.json: updates the IamRoleManagement and PassRoleToLambda resource ARNs to the new role name. - deployment/README.md, README.md, sam-app/README.md: correct IAM: to IAMR: for IAM roles, document the sys:monitor and SVV_TABLE_INFO grants needed for the skill to read other users' query data, and point to the new stack outputs (or deploy.sh's printed commands) instead of assuming a fixed role name. Verified against a live deployment: confirmed the previously-deployed stack's Lambda execution role name via `aws lambda get-function`, and validated the updated SAM template with `sam validate --lint`.
- Correct Step 3 to describe two upload options instead of three, and number the GitHub import steps consistently - Fix skill upload navigation path to Knowledge -> Skills -> Add Skill for both GitHub import and zip upload options - Remove outdated background-mode parenthetical from Step 4 intro
default to background mode. Background execution only when the user explicitly asks, after cluster/database scope is confirmed (SKILL.md Core Rules 10/11, custom agent Section 0) - Treat empty query results as healthy outcomes with a friendly PASS message instead of reporting them as failed - Bump skill to 1.7.0 and custom agent to 1.1.0 with changelogs
…agent with async model Skill v1.8.0 / custom agent v1.3.0. - Remove Capability 4 (Disaster Recovery) and 5 (Incident Detection): both were reference-guidance-only with no live data path. Removed pending a proper CloudWatch MCP server integration. All alarm/DR references scrubbed from instructions and reference docs; orphaned reference files deleted (preserved in the v1.7.0 tag) - Rewrite custom agent Section 0 to 'no scope, no run': custom agents always execute as asynchronous invocations, so scope must come from the invocation prompt; missing scope ends the run with a 'Scope required' report instead of guessing - Document interactive (skill in Chat) vs autonomous (custom agent) use, canceling a background run, downloading the HTML report from Artifacts, the capability-discovery prompt, and version numbers - Fix DevOpsAgentRole trust policy to allow both service/* and agentspace/* SourceArn patterns; raise samconfig capabilities to CAPABILITY_NAMED_IAM
…th detailed breakdowns
…1.8.0 - audit: 98/100 Grade A, 0 criticals, 0 warnings, passed - functional: 94.4% with-skill vs 15.3% without (+79.2%), PARETO_BETTER, passed - trigger: 100% trigger precision, 100% no-trigger precision (2 runs/query), passed - unified report: 0.87 Grade B, passed: true - reword one trigger query to include the skill's operational-review trigger vocabulary
skills/.gitignore's extension allowlist (meant only to restrict what gets zipped for DevOps Agent skill upload) was also blocking the deployment/ directory's infrastructure tooling, which was never part of the skill upload payload to begin with (the documented zip command already excludes deployment/* explicitly). This meant a fresh clone was missing Makefile, run.sh, deploy.sh, build_zip.sh, samconfig.toml, and the test scripts entirely -- causing 'sam build' to fail with 'Makefile not found' on any machine other than the one these files were first created on. Fix: allow all files under deployment/ regardless of extension, while still excluding OS junk and build artifacts (.DS_Store, .aws-sam/, __pycache__/, *.zip) via new explicit patterns.
mcp-proxy's dependency spec allows any mcp>=1.27.1 with no upper bound. The MCP Python SDK just released a breaking v2.0.0 (new stateless 2026-07-28 protocol revision, restructured internal APIs) that mcp-proxy 0.12.0 is not compatible with -- it imports request_ctx from mcp.server.lowlevel.server, which v2 removed and replaced with ServerRequestContext. This meant every fresh 'sam build' from today onward silently packaged a broken combination (mcp==2.0.0 + mcp-proxy==0.12.0), which crashed on Lambda cold start with 'ImportError: cannot import name request_ctx', surfacing to callers as a generic API Gateway 'Internal server error'. Fix: pin mcp==1.29.0 (last 1.x release) and mcp-proxy==0.12.0 explicitly. Verified: rebuilt, redeployed, and confirmed live -- list_clusters now returns HTTP 200 with all 9 clusters/workgroups.
…reorder skill upload options
…IAM permissions, drop version line
…rchitecture diagram
Fix the merge-blocker issues raised in external review of PR aws#30 without changing the skill's documented functionality: - Pin mcp-proxy deps and awslabs.redshift-mcp-server==0.0.29 in both the SAM and plain-CLI build paths, so both resolve identical, reproducible versions (avoids the mcp==2.0.0 ImportError regression and an undocumented 7th tool from 0.0.30+). - Fix invalid SYS_QUERY_DETAIL/SVV_TABLE_INFO column references across the query templates and SKILL.md (spill_local/spill_remote/ bytes_scanned don't exist; SVV_TABLE_INFO.empty is documented as unused). - Add live collection queries for WLM configuration, storage skew, and per-item Spectrum/data-sharing rows so report templates match what's actually collected; fix a second invalid-column bug found along the way (sys_datashare_usage_consumer has no duration column). - Disable API Gateway DataTraceEnabled (was logging full request/ response payloads, including query SQL and results). - Add HTML-escaping guidance for untrusted values substituted into the downloadable HTML report. - Make the API Gateway account-level CloudWatch role conditional so a second stack deploy doesn't overwrite the first's setting. - Fix role-ARN parsing for path-qualified caller roles (split on ":role/" instead of "role/"); document the remaining limitation. - Remove unused Redshift Data API permissions from DevOpsAgentRole (Lambda always executes under its own execution role). - Fix a workflow-ordering contradiction where the Detailed Review workflow called list_databases before the required scope confirmation. - Fix a stale skill-name reference in the custom agent's system prompt, and reconcile its HTML-report policy with the skill's interactive opt-in behavior. - Resolve the llms.txt conflict against upstream/main. - Move the MCP server deployment tooling out of the skill directory into mcp/aws-redshift-mcp-server/, matching the existing mcp/aws-eks-node-diagnostics-mcp/ layout, and update the skill README's paths accordingly.
|
I followed the setup instructions in sample-devops-agent-tools for the redshift-support-specialist, deployed the SAM stack, registered the MCP server, added the skill + custom agent, and exercised all capabilities in an AWS employee sandbox (us-east-1). Both interfaces (Web UI and API/CLI) work end-to-end. |
Contributor
|
Re-reviewed at Blocking —
|
| # | Earlier finding | Resolution |
|---|---|---|
| 1 | Unreproducible deployment, unpinned MCP versions | mcp==1.29.0, mcp-proxy==0.12.0 pinned, with a comment explaining that mcp v2.0.0 removed request_ctx from mcp.server.lowlevel.server and breaks mcp-proxy 0.12.0. Both versions confirmed on PyPI. |
| 2 | Invalid/misleading SQL fields | SVV_TABLE_INFO.empty now carries the AWS "for internal use" caveat instead of being presented as a deleted-row percentage; SYS_QUERY_DETAIL usage reduced to the real spilled_block_local_disk. |
| 3 | Templates demanded uncollected data | Sections that need AWS CLI or CloudWatch access now render as ℹ️ N/A with the reason, rather than requesting data the bundle never returns. |
| 4 | Diagnostic data could be logged or injected | DataTraceEnabled left off with a comment explaining it would capture SQL and results; HTML escaping is now an explicit Core Rule with the character mappings spelled out, plus a SECURITY note in the template. |
| 5 | CFN/IAM account-global side effects and identity assumptions | AWS::ApiGateway::Account retained but documented as account- and Region-wide; the fixed execution-role name justified (Redshift maps it to the database user IAM:redshift-mcp-lambda-execution-role via GetClusterCredentialsWithIAM, so a random suffix would destabilise that mapping); path-qualified caller ARNs documented as unsupported with a manual-grant workaround. |
| 6 | Skill/agent/output contracts contradicted each other | Skill name corrected, and the confirmation-rule conflict resolved explicitly rather than papered over — the system prompt states that Core Rules 10/11's "ask and wait" is impossible in autonomous invocation, so stop-and-request replaces it, never proceed-on-assumption. |
| 7 | Layout and evaluation-evidence violations | Skill directory now contains only supported extensions (md, json, yaml, png, html); .skilleval.yaml reduced to the single STR-016 ignore; implementation scripts live under mcp/ where they belong. |
The llms.txt conflict is also resolved, and both the skill page and the custom-agent page generate.
Two of these I'd call better than what I asked for. The fixed role name I flagged as a defect is actually load-bearing, and the comment explains why — I was wrong to treat it as an oversight. And the Core Rules 10/11 reconciliation confronts the contradiction directly instead of quietly dropping one side.
Also worth recording: @ivanfdez-aws independently deployed the SAM stack, registered the MCP server, added the skill and custom agent, and exercised all capabilities end-to-end in a sandbox through both the Web UI and the API/CLI. That is stronger validation than a diff read, and it covers the runtime behaviour I could not verify statically.
Happy to approve once the strict build passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds the redshift-support-specialist skill (v1.8.0) and its companion custom agent (v1.3.0): Amazon Redshift domain expertise for AWS DevOps Agent, plus a ready-to-use serverless deployment of the
awslabs.redshift-mcp-serverMCP server the skill relies on.Skill capabilities (all live via the six Redshift MCP tools — no AWS CLI access, no CSV uploads, read-only):
What's included:
skills/redshift-support-specialist/— SKILL.md, references, query templates, report templates, thresholds, evals (with committed Agent Skill Eval results)custom-agents/redshift-support-specialist/— system prompt for an autonomous custom agent with a "no scope, no run" guard (custom agents execute as async invocations, so scope must come from the invocation prompt; missing scope ends the run with a "Scope required" report instead of guessing)skills/redshift-support-specialist/deployment/— serverless MCP server deployment (Lambda zip + API Gateway with SigV4/IAM auth, via AWS SAM or plain CLI; no containers, no ECR, no VPC), including a documented fix for the DevOps Agent SigV4 role trust policy (agentspace/*SourceArn is required in addition to the documentedservice/*)llms.txtupdatedType of change
Testing
Agent Skill Eval (results committed under
skills/redshift-support-specialist/evals/):audit: 98/100, Grade A — 0 criticals, 0 warnings, passedfunctional: 94.4% with-skill vs 15.3% without-skill assertion pass rate (+79.2%), classified PARETO_BETTER (quality up, ~50% fewer tokens), passedtrigger: 100% trigger precision and 100% no-trigger precision (6 queries, 2 runs each), passedreport: 0.87, Grade B,"passed": trueManual DevOps Agent testing (live Agent Space + deployed MCP server against real provisioned clusters and Serverless workgroups):
License confirmation