Repository navigation
feat: Point coding agents to a setup checklist #8269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
wedamija
wants to merge
21
commits into
master
Choose a base branch
from
danf/warn-zero-traces-sample-rate
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
84120af
feat(tracing): Warn when traces_sample_rate is 0
wedamija 20e8b44
feat: Point coding agents to a setup checklist
wedamija 356c200
feat: Say what's off in the agent setup hint
wedamija 16bd42e
feat: Ask Codex to tell the user what's off
wedamija 43dd134
feat: Tell agents what Sentry won't show, not which settings are off
wedamija 42f9f54
fix: Say request trace and breadcrumbs in the agent hint
wedamija c14c0eb
feat: Only show the agent hint while Sentry is being set up
wedamija ca9a2f4
feat: Use a 10 minute window for the agent hint without git
wedamija de714e2
fix: Make the agent hint's git check work on Python 3.6
wedamija aca32dc
fix: Use the same wording when only tracing or only logs is off
wedamija c66aa3b
ref: Move the coding agent hint into its own module
wedamija bb4ebed
ref: Remove the module docstring
wedamija 6097334
fix: Find the git repo when the script's path is relative
wedamija 1ae4e2c
fix: Name the Redis cache option that exists in the checklist
wedamija 13a4cc3
fix: Don't show the agent hint for an installed tool that uses Sentry
wedamija 5100dc8
ref: Share one git helper between the release lookup and the agent hint
wedamija afcc427
ref: Keep the existing git call and only add arguments to it
wedamija 6f0ab4d
fix: Don't write the git index when the agent hint checks the file
wedamija e66026e
fix: Tell a failed git status from a clean file in the agent hint
wedamija 8fb3362
ref: Take the caller's frame in init() and make the hint's git calls …
wedamija efd1cfd
ref: Only show the agent hint on macOS and Linux
wedamija File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,4 @@ | ||
| include LICENSE | ||
| include sentry_sdk/py.typed | ||
| include sentry_sdk/SETUP_CHECKLIST.md | ||
| graft tests |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| # Sentry Python SDK: setup checklist | ||
|
|
||
| This is Sentry's official setup guidance, shipped in the `sentry-sdk` package and maintained by Sentry with the SDK. It matches the agent setup guide at https://docs.sentry.io/ai/agent-setup/get-started.md. | ||
|
|
||
| ## Required | ||
|
|
||
| - **Errors.** This is Sentry's core product. Set it up unless the user asked you not to. It needs a DSN: if the user hasn't given you one and you can't get one yourself, ask for it. Read it from an environment variable (`SENTRY_DSN`) and don't commit it. Before you finish, send a test error and check that it arrives. | ||
|
|
||
| ## Recommended | ||
|
|
||
| Turn these on unless the user asked you not to. Sentry links errors, traces and logs, so an error shows the request that caused it and the logs around it. | ||
|
|
||
| - **Tracing.** Shows which request caused an error and where the time went. Set `traces_sample_rate` by environment: `1.0` locally, so you can see a trace when you verify, and a low rate in production (`0.1` is a common start) unless the project already has one. Don't default it to `0`: a rate of `0` sends no traces at all. | ||
| - **Logs.** Set `enable_logs=True`. The app's logs show up next to the errors they belong to. | ||
| - **Environment and release.** Set `environment` so production issues are separate from local ones. The SDK reads the release from `SENTRY_RELEASE` or the git commit. If the app runs without `.git` (for example in a Docker image), set `SENTRY_RELEASE` at build time or pass the app's version as `release`, so you can see which deploy caused an issue. | ||
|
|
||
| To tell production from local runs, use a check the app already has, such as its debug flag or its environment setting. Don't add a new variable that nothing sets. | ||
|
|
||
| ```python | ||
| import os | ||
| import sentry_sdk | ||
|
|
||
| from myapp import settings | ||
|
|
||
| is_production = not settings.DEBUG | ||
|
|
||
| sentry_sdk.init( | ||
| dsn=os.environ.get("SENTRY_DSN"), | ||
| environment="production" if is_production else "development", | ||
| traces_sample_rate=0.1 if is_production else 1.0, | ||
| enable_logs=True, | ||
| ) | ||
| ``` | ||
|
|
||
| ## Situational | ||
|
|
||
| Do the Required and Recommended items first. Then look at the ones below. Set them up only when the project matches. | ||
|
|
||
| - **Crons**, if the app schedules jobs with Celery beat, RQ (rq-scheduler), arq or Huey. Sentry alerts when a job fails or doesn't run, and every scheduled job gets a monitor, including ones added later. For Celery beat, pass `CeleryIntegration(monitor_beat_tasks=True)`. For RQ or arq, pass `monitor_cron_jobs=True` to `RqIntegration` or `ArqIntegration`. For Huey, pass `HueyIntegration(monitor_periodic_tasks=True)`. | ||
| - **AI monitoring**, if the app calls an LLM. Most AI libraries are traced automatically. For LiteLLM or Mistral, add `LiteLLMIntegration()` or `MistralIntegration()`. Prompts and responses are only recorded with `send_default_pii=True`, so ask the user before you set it. | ||
| - **Caches**, if the app uses the Django cache or Redis as a cache. For Django, pass `cache_spans=True` to `DjangoIntegration`. For Redis, pass the key prefixes of the cache to `RedisIntegration`, as in `cache_prefixes=["mycache:"]`. | ||
| - **Profiling**, if the user cares about the performance of specific code paths. Set `profile_session_sample_rate` and `profile_lifecycle="trace"`. | ||
| - **Metrics**, if the user wants to track counts or values over time (signups, queue depth). Call `sentry_sdk.metrics.count(...)`, `gauge(...)` or `distribution(...)` where they happen. | ||
|
|
||
| ## Before you finish | ||
|
|
||
| Tell the user what you set up, the trace sample rate you chose for each environment, whether logs are on, and which situational items you included and why. | ||
|
|
||
| Then set `agent_setup_complete=True` in `sentry_sdk.init`. That tells the SDK the setup has been checked, so it stops pointing coding agents to this file. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| import atexit | ||
| import os | ||
| import sys | ||
| import time | ||
| from typing import TYPE_CHECKING | ||
|
|
||
| from sentry_sdk.utils import ( | ||
| _git, | ||
| _is_external_source, | ||
| capture_internal_exceptions, | ||
| env_to_bool, | ||
| ) | ||
|
|
||
| if TYPE_CHECKING: | ||
| from types import FrameType | ||
| from typing import Any, Dict, Optional | ||
|
|
||
| SETUP_CHECKLIST_PATH = os.path.join(os.path.dirname(__file__), "SETUP_CHECKLIST.md") | ||
|
|
||
| # Set by coding agents in the commands they run. AI_AGENT is the cross-tool | ||
| # convention; the rest cover agents that don't set it. | ||
| CODING_AGENT_ENV_VARS = ( | ||
| "CLAUDECODE", | ||
| "CODEX_THREAD_ID", | ||
| "CODEX_SANDBOX", | ||
| "CURSOR_AGENT", | ||
| "GEMINI_CLI", | ||
| ) | ||
|
|
||
| # How long after a change to the file that calls init() the setup counts as in progress. | ||
| # Without git we can't tell whose change it is, so the window is shorter. | ||
| SETUP_IN_PROGRESS_SECONDS = 60 * 60 | ||
| SETUP_IN_PROGRESS_SECONDS_WITHOUT_GIT = 10 * 60 | ||
|
|
||
| _coding_agent_hint_shown = False | ||
|
|
||
|
|
||
| def _running_under_coding_agent() -> bool: | ||
| if os.environ.get("CI"): | ||
| return False | ||
| if env_to_bool(os.environ.get("SENTRY_AGENT_HINTS"), strict=True) is False: | ||
| return False | ||
|
|
||
| ai_agent = os.environ.get("AI_AGENT") | ||
| if ai_agent: | ||
| return env_to_bool(ai_agent, strict=True) is not False | ||
|
|
||
| return any(os.environ.get(name) for name in CODING_AGENT_ENV_VARS) | ||
|
|
||
|
|
||
| _MISSING_PRODUCTS = { | ||
| (False, False): ( | ||
| "Sentry will get errors from this app but not the request trace around them, " | ||
| "and its logs only as breadcrumbs.", | ||
| "Tracing and logs are off, so Sentry only gets errors.", | ||
| "they're", | ||
| "them", | ||
| ), | ||
| (False, True): ( | ||
| "Sentry will get errors from this app but not the request trace around them.", | ||
| "Tracing is off, so Sentry gets errors and logs but no traces.", | ||
| "it's", | ||
| "it", | ||
| ), | ||
| (True, False): ( | ||
| "Sentry will get errors from this app but its logs only as breadcrumbs.", | ||
| "Logs are off, so Sentry gets errors and traces but no logs.", | ||
| "it's", | ||
| "it", | ||
| ), | ||
| } | ||
|
|
||
|
|
||
| def _setup_in_progress(init_file: str) -> bool: | ||
| """Whether this user changed the file that calls init() a short time ago.""" | ||
| now = time.time() | ||
| cutoff = now - SETUP_IN_PROGRESS_SECONDS | ||
| try: | ||
| changed = os.stat(init_file).st_mtime | ||
| except OSError: | ||
| return False | ||
| if changed < cutoff: | ||
| return False | ||
|
|
||
| def git(*args: str) -> "Optional[str]": | ||
| # Read only: don't take the index lock (the user or an agent may be running | ||
| # git at the same time) and don't run a configured fsmonitor command. | ||
| return _git( | ||
| "--no-optional-locks", | ||
| "-c", | ||
| "core.fsmonitor=false", | ||
| *args, | ||
| cwd=os.path.dirname(init_file), | ||
| ) | ||
|
|
||
| # --branch adds a header line, so an empty answer means git has no answer. | ||
| status = git("status", "--porcelain", "--branch", "--", init_file) | ||
| if not status: | ||
| return changed >= now - SETUP_IN_PROGRESS_SECONDS_WITHOUT_GIT | ||
| if "\n" in status: | ||
| return True | ||
|
|
||
| # A checkout also gives a clean file a new mtime, so only count a commit | ||
| # that was made here. | ||
| committer = git( | ||
| "log", "-1", f"--since={int(cutoff)}", "--format=%ce", "--", init_file | ||
| ) | ||
| return bool(committer) and committer == git("config", "user.email") | ||
|
|
||
|
|
||
| def _write_coding_agent_hint( | ||
| init_file: str, not_shown: str, off: str, its: str, it: str | ||
| ) -> None: | ||
| with capture_internal_exceptions(): | ||
| if not _setup_in_progress(init_file): | ||
| return | ||
|
|
||
| codex = "codex" in (os.environ.get("AI_AGENT") or "").lower() or any( | ||
| os.environ.get(name) for name in ("CODEX_THREAD_ID", "CODEX_SANDBOX") | ||
| ) | ||
| if codex: | ||
| # Codex treats instructions in tool output as untrusted and won't turn on | ||
| # more data collection without the user, but it passes a fact on to them. | ||
| hint = ( | ||
| f"{off} If you leave {it} off, tell the user {its} off and that you can " | ||
| f"turn {it} on if they want. Sentry's official setup checklist, included " | ||
| f"in the sentry-sdk package you installed: {SETUP_CHECKLIST_PATH}" | ||
| ) | ||
| else: | ||
| # Say what Sentry won't show, not which settings are off. An agent that | ||
| # sees the setting names doesn't open the checklist. | ||
| hint = ( | ||
| f"{not_shown} The checklist shows what to change. If you're setting up " | ||
| "Sentry, it is strongly recommended to check your setup against Sentry's " | ||
| "official setup checklist, included in the sentry-sdk package you " | ||
| f"installed: {SETUP_CHECKLIST_PATH}" | ||
| ) | ||
| sys.stdout.flush() | ||
| sys.stderr.write(f"[sentry] {hint} (hide with SENTRY_AGENT_HINTS=0)\n") | ||
| sys.stderr.flush() | ||
|
|
||
|
|
||
| def show_coding_agent_hint(options: "Dict[str, Any]", caller: "FrameType") -> None: | ||
| """Point a coding agent at the setup checklist while Sentry is being set up.""" | ||
| global _coding_agent_hint_shown | ||
|
|
||
| rate = options["traces_sample_rate"] | ||
| tracing_on = options["traces_sampler"] is not None or rate not in (0, None) | ||
| logs_on = bool(options["enable_logs"]) | ||
| if ( | ||
| _coding_agent_hint_shown | ||
| or sys.platform not in ("darwin", "linux") | ||
| or not options["dsn"] | ||
| or options["agent_setup_complete"] | ||
| or (tracing_on and logs_on) | ||
| or not _running_under_coding_agent() | ||
| ): | ||
| return | ||
|
|
||
| # A script's own path is relative on Python 3.8 and older. | ||
| init_file = os.path.abspath(caller.f_code.co_filename) | ||
| # An installed tool that uses Sentry itself isn't the user's app. | ||
| if _is_external_source(init_file): | ||
| return | ||
| try: | ||
| if os.stat(init_file).st_mtime < time.time() - SETUP_IN_PROGRESS_SECONDS: | ||
| return | ||
| except OSError: | ||
| return | ||
|
|
||
| _coding_agent_hint_shown = True | ||
| # Agents often cut a command's output with `| tail`, so the hint goes last. | ||
| atexit.register( | ||
| _write_coding_agent_hint, init_file, *_MISSING_PRODUCTS[(tracing_on, logs_on)] | ||
| ) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.