Contextline

Contextline is a SigNoz-native incident command agent for Kubernetes. It keeps one durable context line from the first alert through live evidence, structured diagnosis, deterministic guardrails, explicit approval, rollback, and verified recovery.

Built for the Agents of SigNoz Hackathon — AI & Agent Observability track.

Why it exists

Incident response loses time when evidence, model reasoning, approvals, and production actions live in separate tools. Contextline makes that chain visible and auditable:

• reads the target Deployment, ReplicaSets, and warning Events through the real Kubernetes API;
• gathers current firing-alert evidence through the official SigNoz MCP server;
• asks an OpenAI-compatible model for a strict, evidence-bounded diagnosis;
• permits only a rollback to an image observed in ReplicaSet history;
• requires a named human approval before any Kubernetes write;
• verifies readiness and image state after execution;
• exports workflow traces, metrics, and logs to SigNoz over OTLP;
• persists every checkpoint in a SHA-256 hash chain and stops automation if the receipt is altered.

There is no Python service and no simulated production evidence. The application is Next.js, React, Node.js, and TypeScript end to end.

Architecture

SigNoz alert → local webhook ┐
                             ├→ Next.js war room → checkpointed workflow
Operator ────────────────────┘
                              ├─ Kubernetes API (evidence + guarded patch)
                              ├─ SigNoz MCP (firing-alert evidence)
                              ├─ OpenAI-compatible model (structured diagnosis)
                              ├─ JSON receipt store (durable hash chain)
                              └─ OTLP gRPC → SigNoz traces / metrics / logs

See docs/architecture.md for the state machine, trust boundaries, and failure modes.

Prerequisites

• Node.js 22+
• Docker Desktop
• kubectl
• an OpenAI-compatible model endpoint (a hosted provider or local Ollama)
• roughly 4 GB of free memory for the local SigNoz and kind stacks

Run the real local stack

Install dependencies:

npm ci

Install SigNoz Foundry using its official installer, then cast the checked-in configuration:

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

The checked-in casting.yaml.lock was generated by Foundry v0.2.16. The cast exposes:

• SigNoz UI: http://localhost:8080
• OTLP gRPC / HTTP: localhost:4317 / localhost:4318
• SigNoz MCP: http://localhost:8000/mcp

On first launch, create the local SigNoz admin account. Under Settings → Service Accounts, create contextline-local, assign the least-privilege signoz-editor role, and generate a key. Editor access is required only for the reproducible dashboard/alert provisioning step.

Prepare the isolated Kubernetes demo:

./scripts/prepare-demo.sh
./scripts/inject-incident.sh

inject-incident.sh changes only contextline-demo/checkout to a deliberately nonexistent image. The healthy nginx:1.27-alpine ReplicaSet remains in history as the only rollback target.

Configure Contextline:

cp .env.example .env.local

Set LLM_API_KEY, LLM_MODEL, and SIGNOZ_API_KEY. A completely local model is also supported through Ollama's OpenAI-compatible endpoint:

LLM_BASE_URL=http://localhost:11434/v1
LLM_API_KEY=ollama-local
LLM_MODEL=qwen3.5:9b
LLM_PROVIDER_NAME=ollama

Create and test an intentional Webhook channel named contextline-local that targets http://host.docker.internal:3000/api/signoz/alerts. The endpoint accepts the standard Alertmanager payload, acknowledges unrelated SigNoz channel tests without creating an incident, deduplicates firing alerts by fingerprint, and opens only alerts marked project=contextline or carrying an explicit Kubernetes namespace/deployment. Set CONTEXTLINE_WEBHOOK_USERNAME and CONTEXTLINE_WEBHOOK_PASSWORD to validate the channel's optional HTTP Basic credentials with constant-time comparison.

Set its exact name in SIGNOZ_NOTIFICATION_CHANNEL, then provision and read back the project assets through the official MCP server:

npm run signoz:configure

This command is safe to rerun: it verifies the selected channel, creates only missing Contextline assets, and fails if the dashboard or any rule cannot be read back. It never invents a notification destination.

Start Contextline:

npm run dev

Open http://localhost:3000, create an incident for namespace contextline-demo and deployment checkout, then advance one checkpoint at a time. Review the proposed action and approve it only after its guardrail checks pass.

Use CONTEXTLINE_ALERT_NAMESPACE and CONTEXTLINE_ALERT_DEPLOYMENT as the explicit fallback target for Contextline-owned alerts whose telemetry rule does not carry Kubernetes labels. Never use these defaults for unrelated alerts.

Workflow and safety

intake
  → gather_evidence
  → diagnose
  → guard_action
  → awaiting_approval
  → execute
  → verify
  → resolved | handoff

Contextline does not run arbitrary commands. Its only write action is a Kubernetes Deployment image patch. That patch is allowed only when:

1. namespace, Deployment, and container match the evidence snapshot;
2. the namespace and Deployment are not protected;
3. the target image was observed in prior ReplicaSet history;
4. the target differs from the current image;
5. a human operator explicitly approves the exact proposal;
6. the incident receipt hash chain is intact;
7. the automation failure budget has not opened the circuit.

Model or dependency failure preserves the last checkpoint. A primary model can fall back once to LLM_DEGRADED_MODEL, but degraded mode is read-only even when that diagnosis succeeds. A verified recovery of the failed primary dependency can restore normal mode; repeated failure enters blackout/handoff mode.

OpenTelemetry and SigNoz

Contextline exports:

• spans for the incident and every workflow stage, with Kubernetes, gen_ai.*, mode, and outcome attributes;
• contextline.incidents, contextline.guardrail.blocks, and contextline.actions counters;
• contextline.stage.duration histogram;
• structured incident checkpoint logs correlated to the active trace.

The SigNoz MCP connection is also a hard readiness signal. Contextline verifies that the server advertises signoz_list_alerts before using it.

See docs/observability.md for the reproducible dashboard, alert rules, Query Builder filters, and live verification queries.

Deployment

Build the standalone image:

docker build -t contextline:local .

deploy/contextline.yaml includes least-privilege namespaced RBAC, a persistent receipt volume, a non-root/read-only container posture, separate liveness and dependency-readiness probes, and a recreate strategy for the single-writer receipt store. Replace its image reference and create the contextline-secrets Secret with llm-api-key, llm-model, signoz-api-key, webhook-username, and webhook-password keys before applying it. Set SIGNOZ_URL to the server-reachable address and SIGNOZ_PUBLIC_URL to the address an operator's browser can open.

Do not deploy this application to an ephemeral serverless runtime: the JSON receipt database requires persistent disk. For a multi-replica production deployment, replace the repository adapter with a transactional database.

Quality checks

npm run validate

This runs ESLint, TypeScript, Vitest, and the production Next.js build. The tests cover the rollback guardrails and tamper-evident receipt chain.

Inspiration and originality

Contextline takes product inspiration—not code—from:

• Backstop: real-cluster remediation, receipts, and conservative guardrails;
• Halo: checkpointed continuity, explicit operational modes, and a focused dark war-room interface.

Contextline is an original implementation centered on SigNoz, OpenTelemetry, Foundry, and the official SigNoz MCP server.

AI disclosure

AI coding assistants were used during research, planning, implementation, and review. All integration claims in this repository are intended to be backed by source, tests, or reproducible live checks; missing credentials are reported as missing rather than replaced with mocks.