diff --git a/README.md b/README.md index 8397dd1..5bfe55d 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,144 @@ +
+ # CodeIntel -**A reusable AI coding-agent skill for deeply understanding software repositories—and preserving that knowledge locally for future work.** +**Persistent codebase memory for AI coding agents.** + +Your agent analyzes a repository once, writes down what it learned, with evidence, in a local `.codeintel/` folder, and keeps those notes current as the code changes.
+Later sessions start from that map instead of re-reading the whole repo. + +[![Validate](https://github.com/sagarwaal/codeintel/actions/workflows/validate.yml/badge.svg)](https://github.com/sagarwaal/codeintel/actions/workflows/validate.yml) +[![Release](https://img.shields.io/github/v/release/sagarwaal/codeintel?color=blue)](https://github.com/sagarwaal/codeintel/releases) +[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) +[![Agent Skills](https://img.shields.io/badge/Agent%20Skills-compatible-8A2BE2)](https://skills.sh/sagarwaal/codeintel/codeintel) +[![GitHub stars](https://img.shields.io/github/stars/sagarwaal/codeintel?style=social)](https://github.com/sagarwaal/codeintel/stargazers) + +Works with **Claude Code** · **OpenAI Codex** · **GitHub Copilot CLI** · **Cursor** · **Gemini CLI** · **Antigravity** · any agent that supports [Agent Skills](https://skills.sh) + +[Quick start](#quick-start) · [See a real example](#see-it-on-a-real-repo) · [Why](#the-problem) · [Install](#installation) · [Usage](#usage) · [FAQ](#faq) + +
+ +--- + +## Quick start + +```text +# Claude Code (other agents: see Installation) +/plugin marketplace add sagarwaal/codeintel +/plugin install codeintel@codeintel +``` + +Then, inside any repository: + +> Use CodeIntel to analyze this repository and build the `.codeintel/` knowledge base. + +That's it. In later sessions the agent finds `.codeintel/` and uses it before planning, debugging, or editing code. After a change, it refreshes only the parts of the knowledge base that the change affected. + +## The problem + +Every new agent session starts from zero. The agent greps around, reads a dozen files, builds a partial picture of the system, and loses it all when the session ends. On a real codebase this leads to: + +- **Wasted tokens and time** spent rediscovering the same architecture every session. +- **Changes that miss part of the system:** a fix in one place that breaks a caller the agent never saw. +- **Confident guesses:** "there are no tests for this" when the agent simply didn't look in the right directory. +- **Code that doesn't match the project's own conventions**, because the agent never found the right reference implementation. + +`CLAUDE.md` / `AGENTS.md` files help, but they're short, written by hand, and go out of date quickly. CodeIntel adds a deeper layer that the agent maintains itself and that cites its sources. + +## Before / after + + + + +
Without CodeIntelWith CodeIntel
-CodeIntel helps an AI agent move beyond a quick README summary. It builds an evidence-based knowledge base covering architecture, components, dependencies, workflows, design conventions, implementation risks, test gaps, and safe feature development. +> *"Add retry support to the HTTP client."* -The generated knowledge lives inside the target project in `.codeintel/` and is excluded from Git by default. Future sessions can use it to navigate the codebase, investigate defects, assess change impact, and implement features without repeatedly rediscovering the entire repository. +The agent searches for "retry", opens 9 files, patches the first plausible spot, and misses that a second code path builds its own connection pool and skips the change. It reports that tests pass after running the one test file it found. -## Why + -AI coding agents can lose useful context between sessions and may make incomplete changes when they do not understand the wider system. Codeintel gives them a structured, maintainable map of the codebase—with explicit evidence, uncertainty, and refresh guidance. +> *"Add retry support to the HTTP client."* -It is designed to answer questions such as: +The agent reads `.codeintel/agents/TASK_ROUTING.md`, which points it to the owning component, the [reference pattern](#what-it-creates) to copy, **both** code paths that build connection pools, the contract callers depend on, and the exact focused test command. It makes one coherent change and then refreshes the affected knowledge. -- Where does this behavior belong? -- Which components, interfaces, and workflows will this change affect? -- Which implementation is the best existing reference pattern? -- What invariants and contracts must be preserved? -- Which behaviors lack meaningful tests? -- Is a suspected issue a confirmed defect or only a hypothesis? -- What should be rechecked after a change? +
+ +## See it on a real repo + +**[`examples/requests/`](examples/requests/)** holds a complete knowledge base that CodeIntel generated for [psf/requests](https://github.com/psf/requests) (v2.32.3). It is what your agent would write into `.codeintel/`, committed here so you can browse it without installing anything. + +Good places to start: + +- [`INDEX.md`](examples/requests/.codeintel/INDEX.md): the entry point and task router +- [`ARCHITECTURE.md`](examples/requests/.codeintel/ARCHITECTURE.md): the architecture as implemented, with a diagram +- [`workflows/`](examples/requests/.codeintel/workflows/get-request-trace.md): `requests.get()` traced end to end +- [`TEST_COVERAGE_GAPS.md`](examples/requests/.codeintel/TEST_COVERAGE_GAPS.md): behavioral gaps, not just line coverage +- [`KNOWN_ISSUES.md`](examples/requests/.codeintel/KNOWN_ISSUES.md): findings with evidence and confidence levels + +## How it compares + +| | `CLAUDE.md` / `AGENTS.md` | Editor code index / RAG | **CodeIntel** | +|---|---|---|---| +| What it holds | Short, hand-written instructions | Embeddings of code chunks | Architecture, components, dependencies, workflows, contracts, test gaps, findings | +| Who writes it | You | The tool | Your agent, following a strict evidence protocol | +| Answers "what breaks if I change X?" | Rarely | Partially (by text similarity) | Yes: dependency and impact maps | +| Distinguishes fact from guess | No | No | Yes: `VERIFIED` / `INFERRED` / `SUSPECTED` / `STALE` | +| Stays current | Manual | Re-indexes text | Incremental refresh: detects drift since the last analyzed commit and patches only affected docs | +| Readable by humans | Yes | No | Yes: plain Markdown + Mermaid | +| Lock-in | None | Tool-specific | None: one skill folder works across 6+ agents | + +CodeIntel works alongside these. It can add a short discovery hook to your existing `AGENTS.md` / `CLAUDE.md` (only with your permission), so every future session knows to read `.codeintel/` first. ## What it creates -The skill adapts its output to the size and structure of the repository. A typical knowledge base may look like: +The skill scales its output to the repository. A typical knowledge base: ```text your-project/ ├── .codeintel/ # Local, Git-ignored engineering knowledge -│ ├── INDEX.md +│ ├── INDEX.md # entry point + task routing │ ├── PROJECT_PROFILE.md -│ ├── ARCHITECTURE.md +│ ├── ARCHITECTURE.md # as implemented, with Mermaid diagrams │ ├── DIRECTORY_MAP.md │ ├── COMPONENT_CATALOG.md -│ ├── DEPENDENCY_MAP.md +│ ├── DEPENDENCY_MAP.md # blast radius for changes │ ├── DATA_FLOW.md │ ├── API_AND_CONTRACTS.md -│ ├── DESIGN_PRINCIPLES.md -│ ├── TEST_STRATEGY.md +│ ├── DESIGN_PRINCIPLES.md # conventions + reference implementations +│ ├── TEST_STRATEGY.md # real test commands │ ├── TEST_COVERAGE_GAPS.md -│ ├── KNOWN_ISSUES.md +│ ├── KNOWN_ISSUES.md # evidence-backed findings │ ├── FEATURE_IMPLEMENTATION_GUIDE.md -│ ├── agents/ +│ ├── agents/ # QUICK_START, TASK_ROUTING, REVIEW_CHECKLIST │ ├── components/ │ ├── workflows/ │ ├── findings/ -│ └── manifests/ +│ └── manifests/ # analysis_state.json: commit checkpoint + claims ├── src/ -├── tests/ └── ... ``` -The exact files are chosen based on repository complexity; the skill should not create empty boilerplate just to match a template. +The skill picks which files to create based on how complex the repository is. It does not create empty boilerplate just to fill a template. ## Capabilities -- **Repository reconnaissance:** identify entry points, languages, frameworks, packages, tests, integrations, and project instructions. -- **Architecture reconstruction:** trace startup, critical workflows, data flows, interfaces, and architectural boundaries. -- **Component intelligence:** document responsibilities, contracts, dependencies, invariants, side effects, and testing approaches. -- **Change-impact analysis:** map important relationships and identify downstream consumers and likely blast radius. -- **Evidence-backed audits:** record correctness defects, cross-component inconsistencies, maintainability risks, and uncertainty. -- **Test intelligence:** map behavior to tests and identify meaningful behavioral coverage gaps—not just line coverage. -- **Project-specific design guidance:** derive conventions from real implementations and distinguish them from recommendations. -- **Security, performance, and reliability review:** identify relevant risks while separating confirmed findings from hypotheses. -- **Incremental refresh:** update affected knowledge as the repository changes rather than regenerating everything after every task. -- **Controlled specialist subagent layer:** delegate independent read-only investigations (`repository_mapper`, `behavior_tracer`, `change_impact_analyst`, `test_quality_analyst`, `defect_consistency_auditor`, `knowledge_reconciler`) using a shared evidence model (`agent-result.json` and `claim.json`), reconciled by a single controlled Knowledge Writer—or run the same roles sequentially when subagents are unavailable. -- **Agent-friendly navigation:** provide quick-start guidance and task routing so future sessions can find the right code and tests. +- **Repository reconnaissance:** entry points, languages, frameworks, packages, tests, integrations, and existing project instructions. +- **Architecture reconstruction:** startup, critical workflows, data flows, interfaces, and boundaries, documented as they actually exist in the code. +- **Component intelligence:** responsibilities, contracts, dependencies, invariants, side effects, and how each component is tested. +- **Change-impact analysis:** downstream consumers and likely blast radius. +- **Evidence-backed audits:** correctness defects, cross-component inconsistencies, and maintainability risks, each with a confidence level. +- **Test intelligence:** maps behavior to tests and finds meaningful behavioral gaps, not just line coverage. +- **Project-specific design guidance:** conventions taken from real implementations and kept separate from recommendations. +- **Security, performance, and reliability review:** keeps confirmed findings separate from hypotheses. +- **Incremental refresh:** a two-phase drift check and patch, keyed to the last analyzed commit. +- **Optional specialist subagents:** six read-only roles (`repository_mapper`, `behavior_tracer`, `change_impact_analyst`, `test_quality_analyst`, `defect_consistency_auditor`, `knowledge_reconciler`) share an evidence model (`agent-result.json` and `claim.json`), and a single controlled Knowledge Writer reconciles their results. If the agent doesn't support subagents, the same roles run one after another. +- **Agent-friendly navigation:** quick-start and task-routing docs so future sessions land on the right code and tests. ## Installation -The repository is a plugin for several coding agents. Every agent reads the same `skills/codeintel/` folder. If you use more than one agent, install it in each one. +This repository is a plugin for several coding agents, and every agent reads the same `skills/codeintel/` folder. If you use more than one agent, install it in each. ### Claude Code @@ -123,7 +193,7 @@ Copy `skills/codeintel/` into your agent's skills directory (for example `~/.cla ## Usage -The skill is named **`codeintel`**. Invoke it explicitly where your agent supports it (for example `/codeintel`), or just describe a task like one of these and the agent will pick it up from the skill description. +The skill is named **`codeintel`**. Invoke it explicitly where your agent supports it (for example `/codeintel`), or describe a task like one of these and the agent will pick it up from the skill description. ### 1. Deep initial analysis @@ -152,24 +222,62 @@ The skill is named **`codeintel`**. Invoke it explicitly where your agent suppor 7. **Incremental, not repetitive.** Refresh affected knowledge based on the change set and dependency relationships. 8. **Adapt to the repository.** Do not force a fixed architecture or create empty documents for small projects. +## FAQ + +
+Does CodeIntel send my code anywhere? + +No. CodeIntel is a set of instructions (a skill) plus small shell scripts. Your existing coding agent does the analysis with the access it already has, and the output is written only to `.codeintel/` inside your repository. Nothing is uploaded by CodeIntel. +
+ +
+How is this different from just writing a good CLAUDE.md / AGENTS.md? + +Keep those files. They're the right place for short, hand-written rules. CodeIntel produces the deep, generated layer underneath them: component docs, dependency and impact maps, workflow traces, test gaps, and findings, each with evidence and a confidence label. It also detects when that knowledge has gone stale. +
+ +
+Will it modify my source code? + +Not during analysis or refresh. The skill only writes inside `.codeintel/`, plus one local Git exclude rule (see below). It asks before adding a discovery hook to `AGENTS.md` / `CLAUDE.md`, and it changes application code only when you ask it to implement something. +
+ +
+How much does the first analysis cost? + +It depends on the size of the repository and your agent. Deep Analysis is a one-time investment. After that, most work runs in the cheaper **Workflow Grounding** and **incremental refresh** modes, which read only the relevant knowledge files and the files that changed. For large repos and monorepos, the skill works in phases, starts with high-impact areas, and records partial coverage honestly. +
+ +
+Should I commit .codeintel/? + +By default, no. It's excluded via `.git/info/exclude` so it creates no tracked change. Some teams choose to share it. If you do, review it first (see [Git exclusion and privacy](#git-exclusion-and-privacy)). +
+ +
+How does it stay up to date? + +`.codeintel/manifests/analysis_state.json` records the last analyzed commit and which documents depend on which source files. A refresh first runs a read-only drift check (Phase A), then revalidates and patches only the affected claims (Phase B). The checkpoint moves forward only if validation passes. [`scripts/codeintel-refresh.sh`](scripts/codeintel-refresh.sh) handles the parts that can be done deterministically. +
+ ## Git exclusion and privacy -By default the skill adds this rule to the clone-local `.git/info/exclude`, so it creates no tracked change in your repository. It uses the shared `.gitignore` instead only if you ask, or if the project already ignores similar tooling directories there: +By default the skill adds this rule to the clone-local `.git/info/exclude`, so it creates no tracked change in your repository. It writes to the shared `.gitignore` instead only if you ask, or if the project already ignores similar tooling directories there: ```gitignore # Local AI-generated code intelligence and repository analysis /.codeintel/ ``` -It verifies the ignore rule when Git is available. If files in the directory are already tracked, ignore rules alone will not untrack them; the skill must report that rather than silently changing tracking state. +It verifies the ignore rule when Git is available. If files in the directory are already tracked, ignore rules alone will not untrack them; in that case the skill reports the problem instead of silently changing what Git tracks. **Important:** Git-ignored does not mean encrypted, private, or secure. The knowledge base may contain sensitive architectural information. Do not store credentials, tokens, private keys, sensitive production data, or secrets in it. Review the generated knowledge before sharing it or uploading it elsewhere. -Because the knowledge directory is ignored, it is not automatically shared with teammates or synchronized between machines through normal Git operations. Back it up or share it separately only when appropriate. +Because the knowledge directory is ignored, normal Git operations won't share it with teammates or sync it between machines. Back it up or share it separately only when appropriate. ## Limitations -Codeintel is an analysis workflow, not a guarantee of exhaustive correctness. Results depend on repository access, available tools, execution time, static-analysis capabilities, test setup, and the parts of the code actually inspected. +CodeIntel is an analysis workflow, not a guarantee of exhaustive correctness. Results depend on repository access, available tools, execution time, static-analysis capabilities, test setup, and which parts of the code were actually inspected. Dynamic dispatch, reflection, generated code, runtime registration, external consumers, and undocumented production behavior may limit certainty. The skill is instructed to record these limitations rather than claim complete coverage. @@ -183,13 +291,15 @@ Dynamic dispatch, reflection, generated code, runtime registration, external con ├── .cursor-plugin/ # Cursor plugin ├── .antigravity-plugin/ # Antigravity plugin ├── gemini-extension.json # Gemini CLI extension +├── examples/ +│ └── requests/ # sample knowledge base generated for psf/requests ├── scripts/ │ ├── validate.sh # manifest, skill, and schema v2 validation │ └── codeintel-refresh.sh # drift detection, schema migration, and refresh engine ├── tests/ │ └── test-refresh.sh # 16-scenario automated fixture refresh, evidence & semantic regression test suite ├── skills/ -│ └── codeintel/ # the skill (invoke as /codeintel) +│ └── codeintel/ # the skill (invoke as /codeintel) │ ├── SKILL.md │ ├── agents/ # 6 read-only specialist role protocols & reconciler │ ├── references/ # architecture, change-protocol, evidence-model, impact-analysis, etc. @@ -209,9 +319,11 @@ Useful contributions include: - Improving analysis workflows and evidence quality. - Adding language- or framework-specific guidance without making the core skill overly prescriptive. - Improving incremental refresh and stale-knowledge detection. -- Adding realistic examples and validation tests. +- Adding realistic examples and validation tests. **Ran CodeIntel on an interesting open-source repo? Open a PR adding it to `examples/`.** - Reporting cases where the skill generates noisy, redundant, or misleading knowledge. +If CodeIntel saved you a session of rediscovering a codebase, **a ⭐ helps other developers find it.** + ## License [MIT](LICENSE) diff --git a/assets/social-preview.png b/assets/social-preview.png new file mode 100644 index 0000000..2fd3898 Binary files /dev/null and b/assets/social-preview.png differ diff --git a/examples/requests/.codeintel/API_AND_CONTRACTS.md b/examples/requests/.codeintel/API_AND_CONTRACTS.md new file mode 100644 index 0000000..43a2eb2 --- /dev/null +++ b/examples/requests/.codeintel/API_AND_CONTRACTS.md @@ -0,0 +1,48 @@ +# API & Contracts: `psf/requests` + +## Top-Level Public Interface (`requests.*`) + +The public interface exposed at `requests.*` maps directly to `requests.api`: + +```python +requests.request(method, url, **kwargs) +requests.get(url, params=None, **kwargs) +requests.post(url, data=None, json=None, **kwargs) +requests.put(url, data=None, **kwargs) +requests.patch(url, data=None, **kwargs) +requests.delete(url, **kwargs) +requests.head(url, **kwargs) +requests.options(url, **kwargs) +``` + +### Accepted `**kwargs` Parameters +- `params`: Dictionary, list of tuples or bytes to append to query string. +- `data`: Dictionary, list of tuples, bytes, or file-like object to send in the body. +- `json`: A JSON-serializable python object to send in the body. +- `headers`: Dictionary of HTTP Headers to send with the `Request`. +- `cookies`: Dict or CookieJar object to send with the `Request`. +- `files`: Dictionary of `'name': file-like-objects` (or `('filename', fileobj)`) for multipart encoding. +- `auth`: Auth tuple or callable object for HTTP authentication. +- `timeout`: Float describing timeout of the request in seconds, or a `(connect_timeout, read_timeout)` tuple. +- `allow_redirects`: Boolean. Enable/disable GET/OPTIONS/POST/etc. redirection handling. +- `proxies`: Dictionary mapping protocol to the URL of the proxy. +- `verify`: Boolean (whether to verify SSL cert) or string (path to CA bundle). +- `stream`: Boolean. If `False`, response content will be immediately downloaded. +- `cert`: String (path to SSL client cert file) or tuple `('cert', 'key')`. + +--- + +## Adapter Contract (`requests.adapters.BaseAdapter`) + +Any transport adapter mounted via `session.mount(prefix, adapter)` must adhere to: + +```python +class BaseAdapter: + def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): + """Sends PreparedRequest object. Returns Response object.""" + raise NotImplementedError + + def close(self): + """Cleans up adapter-specific resources.""" + raise NotImplementedError +``` diff --git a/examples/requests/.codeintel/ARCHITECTURE.md b/examples/requests/.codeintel/ARCHITECTURE.md new file mode 100644 index 0000000..7f8f743 --- /dev/null +++ b/examples/requests/.codeintel/ARCHITECTURE.md @@ -0,0 +1,55 @@ +# Architecture: `psf/requests` + +Requests is organized into four distinct layers: + +1. **Facade Layer (`requests.api` & `requests.__init__`)**: Convenience functions (`get`, `post`, `put`, `delete`, `head`, `options`, `patch`, `request`) that manage ephemeral `Session` instances. +2. **Session & Orchestration Layer (`requests.sessions`)**: Maintains cookies across requests, stores auth/header defaults, resolves redirects (`SessionRedirectMixin`), and routes requests to transport adapters based on URL prefixes. +3. **Data Model Layer (`requests.models`)**: + - `Request`: Raw request configuration supplied by caller. + - `PreparedRequest`: Immutable request payload (headers formatted, body encoded, URL canonicalized with query parameters, cookies attached) ready for socket dispatch. + - `Response`: Encapsulates status code, headers, cookies, raw byte streaming (`iter_content`), text decoding (`text`), and redirect history. +4. **Transport Layer (`requests.adapters` & `urllib3`)**: + - `BaseAdapter`: Abstract contract for transport handlers. + - `HTTPAdapter`: Concrete adapter that manages `urllib3.PoolManager` and `urllib3.ProxyManager`, translates `PreparedRequest` into `urllib3` calls, and packages the result into a `Response`. + +--- + +## Architectural Workflow Diagram + +```mermaid +flowchart TD + User([Caller / Application Code]) -->|requests.get / post| Facade[requests.api] + Facade -->|instantiates ephemeral| Session[requests.sessions.Session] + User -.->|or uses persistent| Session + + Session -->|1. prepare_request| RequestModel[requests.models.Request] + RequestModel -->|prepare| PrepReq[requests.models.PreparedRequest] + + Session -->|2. get_adapter for URL prefix| AdapterRegistry{Adapter Mount Map} + AdapterRegistry -->|http:// or https://| HTTPAdapter[requests.adapters.HTTPAdapter] + + HTTPAdapter -->|3. send PreparedRequest| PoolMgr[urllib3.PoolManager / ProxyManager] + PoolMgr -->|4. socket I/O| RemoteServer[(Target HTTP/HTTPS Host)] + RemoteServer -->|5. raw urllib3 response| PoolMgr + PoolMgr -->|urllib3 HTTPResponse| HTTPAdapter + + HTTPAdapter -->|6. build_response| ResponseModel[requests.models.Response] + ResponseModel -->|7. attach history / resolve redirects| Session + Session -->|8. return Response| User +``` + +--- + +## Key Architectural Decisions & Evidence + +### 1. Ephemeral vs. Persistent Sessions +- **Evidence:** `src/requests/api.py:14-64` +- Calling `requests.get()` opens an ephemeral `with sessions.Session() as session:` block. Each top-level call builds and tears down a connection pool unless the caller explicitly uses `requests.Session()`. + +### 2. URL Prefix Adapter Dispatch +- **Evidence:** `src/requests/sessions.py:734-754` (`Session.get_adapter`) +- Sessions store an ordered dictionary of adapters mounted to prefixes (`self.adapters = CaseInsensitiveDict()`). Standard setup mounts `https://` and `http://` to default `HTTPAdapter` instances. Custom adapters can be mounted for custom schemes or specific domains. + +### 3. Preparation Pipeline +- **Evidence:** `src/requests/models.py:277-505` (`PreparedRequest.prepare`) +- Preparation occurs sequentially: `prepare_method`, `prepare_url`, `prepare_headers`, `prepare_cookies`, `prepare_body`, `prepare_auth`, `prepare_hooks`. Each step mutates only the `PreparedRequest` instance. diff --git a/examples/requests/.codeintel/COMPONENT_CATALOG.md b/examples/requests/.codeintel/COMPONENT_CATALOG.md new file mode 100644 index 0000000..64b229a --- /dev/null +++ b/examples/requests/.codeintel/COMPONENT_CATALOG.md @@ -0,0 +1,29 @@ +# Component Catalog: `psf/requests` + +### 1. `Session` (`requests.sessions.Session`) +- **Location:** `src/requests/sessions.py:348` +- **Role:** Central orchestrator for request execution and client-side state (cookies, headers, authentication, adapter registry). +- **Key Methods:** `request()`, `send()`, `mount()`, `prepare_request()`, `resolve_redirects()`. +- **Contracts:** Inherits from `SessionRedirectMixin`. Must be closed via `.close()` or used as a context manager (`with Session() as s:`) to properly dispose underlying connection pools. + +### 2. `PreparedRequest` (`requests.models.PreparedRequest`) +- **Location:** `src/requests/models.py:277` +- **Role:** Holds the fully evaluated request payload. Prepared requests contain canonicalized headers, serialized body streams, and full URLs. +- **Key Methods:** `prepare()`, `prepare_url()`, `prepare_headers()`, `prepare_body()`, `prepare_auth()`, `copy()`. +- **Invariants:** Does not execute network I/O. Safe to clone and inspect before execution. + +### 3. `Response` (`requests.models.Response`) +- **Location:** `src/requests/models.py:601` +- **Role:** Encapsulates server HTTP response metadata and body content. +- **Key Attributes / Methods:** `status_code`, `headers`, `content` (raw bytes), `text` (decoded str), `json()`, `iter_content()`, `raise_for_status()`. +- **Contracts:** `response.raw` holds the raw `urllib3.response.HTTPResponse` object. Accessing `.content` caches all bytes in memory. + +### 4. `HTTPAdapter` (`requests.adapters.HTTPAdapter`) +- **Location:** `src/requests/adapters.py:54` +- **Role:** Bridges `requests` with `urllib3.PoolManager`. Manages socket connection pools, TLS configurations (`cert`, `verify`), and translates `urllib3` exceptions into `requests.exceptions`. +- **Key Methods:** `send()`, `build_response()`, `get_connection()`, `close()`. +- **Invariants:** Owns connection pool lifecycles. Can be subclassed to implement custom retry logic, circuit breakers, or custom transports. + +### 5. `CaseInsensitiveDict` (`requests.structures.CaseInsensitiveDict`) +- **Location:** `src/requests/structures.py:13` +- **Role:** Specialized mapping structure for HTTP headers where lookup is case-insensitive, while preserving the casing of original keys when iterated over. diff --git a/examples/requests/.codeintel/DATA_FLOW.md b/examples/requests/.codeintel/DATA_FLOW.md new file mode 100644 index 0000000..0c6fe72 --- /dev/null +++ b/examples/requests/.codeintel/DATA_FLOW.md @@ -0,0 +1,53 @@ +# Data Flow & Lifecycle: `psf/requests` + +## The Request Lifecycle + +```text +1. Caller Invocation + requests.get(url, params=..., headers=...) + │ + ▼ +2. Ephemeral Session Creation (in requests.api) + session = Session() + session.request('GET', url, ...) + │ + ▼ +3. Request Construction & Preparation + req = Request(method='GET', url=url, headers=headers, files=..., data=..., json=..., params=...) + prep = session.prepare_request(req) + [prep.prepare_url: parameters encoded, idna resolved] + [prep.prepare_headers: CaseInsensitiveDict initialized] + [prep.prepare_auth: HTTPBasicAuth or custom auth applied] + [prep.prepare_cookies: Session cookies merged with Request cookies] + │ + ▼ +4. Adapter Resolution + adapter = session.get_adapter(url=prep.url) # matches "http://" or "https://" prefix + │ + ▼ +5. Adapter Execution + resp = adapter.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies) + [urllib3 connection pool checked out] + [Low-level urllib3.response.HTTPResponse received] + [requests.models.Response constructed via adapter.build_response()] + │ + ▼ +6. Redirect Resolution (if resp.is_redirect and allow_redirects=True) + generator = session.resolve_redirects(resp, prep, stream=stream, ...) + [Loops through 301/302/303/307/308 responses up to max_redirects (default 30)] + [History appended to final response.history] + │ + ▼ +7. Hook Dispatch & Return + resp = dispatch_hook('response', prep.hooks, resp) + return resp +``` + +## Response Body Consumption Flow + +1. **Eager Mode (`stream=False`, default):** + - `adapter.build_response()` does not read body immediately, but `Session.send()` does not close it. + - Calling `response.content` invokes `response._content = b"".join(response.iter_content(CONTENT_CHUNK_SIZE))`, eagerly loading all bytes and releasing the connection back to the `urllib3` pool. +2. **Streaming Mode (`stream=True`):** + - Socket connection stays open and attached to `response.raw`. + - Caller must iterate via `response.iter_content()` or `response.iter_lines()`, or explicitly call `response.close()`. diff --git a/examples/requests/.codeintel/DEPENDENCY_MAP.md b/examples/requests/.codeintel/DEPENDENCY_MAP.md new file mode 100644 index 0000000..7d7ea3b --- /dev/null +++ b/examples/requests/.codeintel/DEPENDENCY_MAP.md @@ -0,0 +1,31 @@ +# Dependency Map & Blast Radius: `psf/requests` + +## Internal Coupling Graph + +```text +requests.api + └── requests.sessions.Session + ├── requests.models.Request + │ └── requests.models.PreparedRequest + ├── requests.adapters.HTTPAdapter (implements BaseAdapter) + │ └── urllib3.PoolManager / ProxyManager + ├── requests.cookies.RequestsCookieJar + └── requests.auth (AuthBase, HTTPBasicAuth) +``` + +## Blast Radius Analysis + +### High Blast Radius Components +1. **`requests.models.PreparedRequest`** + - **Impact Level:** 🚨 Critical + - **Reason:** Every adapter, hook, and authentication handler operates on `PreparedRequest`. Any modification to its internal attributes (`.url`, `.headers`, `.body`, `.hooks`) affects all transport adapters and session pipelines. +2. **`requests.sessions.Session.send`** + - **Impact Level:** 🚨 Critical + - **Reason:** Core dispatch pipeline responsible for resolving adapter targets, attaching cookies, and executing redirect loops. Changes here affect every HTTP request in the library. +3. **`requests.adapters.HTTPAdapter.send`** + - **Impact Level:** 🚨 Critical + - **Reason:** Direct bridge to `urllib3`. Handles exception translation, timeout mapping, proxy configuration, and certificate validation. + +### Low Blast Radius Components +1. **`requests.status_codes`**: Static lookup dict; modifying entries affects only status symbol resolution (`codes.ok`). +2. **`requests.help`**: Informational module printing environment diagnostics; has zero runtime side effects on HTTP operations. diff --git a/examples/requests/.codeintel/DESIGN_PRINCIPLES.md b/examples/requests/.codeintel/DESIGN_PRINCIPLES.md new file mode 100644 index 0000000..7732db9 --- /dev/null +++ b/examples/requests/.codeintel/DESIGN_PRINCIPLES.md @@ -0,0 +1,32 @@ +# Design Principles & Reference Implementations: `psf/requests` + +## Core Design Principles + +1. **Explicit Ergonomics over Obscure Configuration:** The API defaults to what 99% of developers need (automatic redirect following on GET, JSON decoding helper, connection pooling, cert verification enabled by default). +2. **Defensive Translation of Underlying Exceptions:** Never leak `urllib3` exceptions directly to the user. Every `urllib3` error must be caught in `HTTPAdapter.send` and re-raised as a corresponding `requests.exceptions.RequestException` subclass. +3. **Pluggable Architecture via Adapters:** Transport-level variations (retries, UNIX domain sockets, mock backends) are extended via `HTTPAdapter` subclasses mounted to URL prefixes, keeping the `Session` orchestrator clean. + +--- + +## Canonical Reference Patterns + +### Pattern 1: Translation of Transport Exceptions +- **Reference:** `src/requests/adapters.py:510-600` +- **Why it is a good reference:** Demonstrates the canonical pattern for catching low-level socket/HTTP client errors and converting them into high-level user-friendly exceptions without losing traceback context. + +### Pattern 2: Custom Authentication via `AuthBase` +- **Reference:** `src/requests/auth.py:72-102` (`HTTPBasicAuth`) +- **Implementation Guide:** + ```python + from requests.auth import AuthBase + + class CustomTokenAuth(AuthBase): + def __init__(self, token): + self.token = token + + def __call__(self, r): + # Mutate PreparedRequest headers + r.headers['X-Custom-Token'] = self.token + return r + ``` +- **Why it is a good reference:** Shows how authentication handlers hook cleanly into `PreparedRequest` before network dispatch without tight coupling to the transport adapter. diff --git a/examples/requests/.codeintel/DIRECTORY_MAP.md b/examples/requests/.codeintel/DIRECTORY_MAP.md new file mode 100644 index 0000000..b6def49 --- /dev/null +++ b/examples/requests/.codeintel/DIRECTORY_MAP.md @@ -0,0 +1,19 @@ +# Directory Map: `psf/requests` + +| Path | Purpose | Key Files & Entry Points | +| :--- | :--- | :--- | +| `src/requests/` | Core package source root. | `__init__.py` | +| `src/requests/api.py` | Top-level convenience functions (`get`, `post`, `request`). | `request()`, `get()`, `post()` | +| `src/requests/sessions.py` | State persistence, connection reuse, redirect loops, adapter mounting. | `Session`, `SessionRedirectMixin` | +| `src/requests/adapters.py` | Transport layer mapping `PreparedRequest` to `urllib3.PoolManager`. | `BaseAdapter`, `HTTPAdapter` | +| `src/requests/models.py` | Request/Response data models and data serialization. | `Request`, `PreparedRequest`, `Response` | +| `src/requests/auth.py` | Authentication handlers. | `AuthBase`, `HTTPBasicAuth`, `HTTPDigestAuth` | +| `src/requests/cookies.py` | CookieJar compatibility, extraction, and merging helpers. | `RequestsCookieJar`, `cookiejar_from_dict` | +| `src/requests/exceptions.py` | Exception hierarchy rooted in `RequestException`. | `RequestException`, `HTTPError`, `ConnectionError`, `Timeout` | +| `src/requests/structures.py` | Internal specialized data structures. | `CaseInsensitiveDict`, `LookupDict` | +| `src/requests/utils.py` | Common utility functions (URL quoting, header parsing, proxy discovery). | `get_environ_proxies`, `super_len`, `urldefragauth` | +| `src/requests/compat.py` | Python standard library compatibility shims. | `urlparse`, `str`, `bytes` imports | +| `src/requests/status_codes.py` | HTTP status code lookup dict (`codes.ok`, `codes.not_found`). | `_codes`, `codes` | +| `tests/` | Pytest test suite covering unit and integration scenarios. | `test_requests.py`, `conftest.py` | +| `tests/testserver/` | Threaded socket server used to test HTTP behavior without internet access. | `server.py` | +| `tests/certs/` | Test certificates (valid, expired, and mTLS certificates). | `certs/valid/`, `certs/expired/`, `certs/mtls/` | diff --git a/examples/requests/.codeintel/FEATURE_IMPLEMENTATION_GUIDE.md b/examples/requests/.codeintel/FEATURE_IMPLEMENTATION_GUIDE.md new file mode 100644 index 0000000..f349a51 --- /dev/null +++ b/examples/requests/.codeintel/FEATURE_IMPLEMENTATION_GUIDE.md @@ -0,0 +1,34 @@ +# Feature Implementation Guide: `psf/requests` + +Use this guide when adding features or fixing bugs in `psf/requests` to adhere to project invariants. + +--- + +## 1. Locate Owning Layer +- Adding a high-level user API parameter (e.g. new keyword arg) ➔ `requests.api`, `requests.sessions.Session.request`, and `requests.models.Request`. +- Modifying how headers/body/params are prepared ➔ `requests.models.PreparedRequest.prepare_*`. +- Modifying transport, retries, SSL verification, or proxies ➔ `requests.adapters.HTTPAdapter`. +- Adding auth mechanism ➔ `requests.auth.AuthBase`. + +--- + +## 2. Parameter Propagation Invariant +When adding any new parameter to `requests.get()` or `requests.post()`: +1. Add it to `requests/api.py` in the appropriate wrapper function and propagate to `session.request()`. +2. Update `Session.request()` in `src/requests/sessions.py` to accept the parameter. +3. Pass it to `Session.prepare_request()` or `Session.send()` depending on whether it affects the payload or transport. +4. If it affects the transport, ensure `HTTPAdapter.send()` in `src/requests/adapters.py` accepts and processes it. + +--- + +## 3. Preserving Backwards Compatibility +- Never remove or re-order positional arguments on public methods. +- Default all new options to values that preserve existing behavior (e.g. `feature_enabled=False` or `None`). +- Return types must remain `Response` instances. + +--- + +## 4. Testing Checklist +- Add unit tests in `tests/test_requests.py` verifying both default behavior and new option behavior. +- Ensure all tests run offline without remote internet access. +- Run `pytest` locally to confirm 0 regressions across the existing test suite. diff --git a/examples/requests/.codeintel/INDEX.md b/examples/requests/.codeintel/INDEX.md new file mode 100644 index 0000000..aea0140 --- /dev/null +++ b/examples/requests/.codeintel/INDEX.md @@ -0,0 +1,54 @@ +# CodeIntel Knowledge Base: `psf/requests` + +> **Repository:** [psf/requests](https://github.com/psf/requests) +> **Analyzed Version:** `v2.32.3` (`0e322af87745eff34caffe4df68456ebc20d9068`) +> **Schema Version:** 2 +> **Analysis Mode:** Deep Analysis (Offline showcase) + +Welcome to the CodeIntel knowledge base for **Requests: HTTP for Humans™**. This directory is an evidence-backed architectural and engineering map generated to ground AI coding agents and engineers navigating, extending, or debugging `requests`. + +--- + +## 🗺️ Key Documents & Entry Points + +| Document | Purpose & Key Takeaways | +| :--- | :--- | +| **[`PROJECT_PROFILE.md`](PROJECT_PROFILE.md)** | Core domain overview, ecosystem role, target environments, and external runtime dependencies (`urllib3`, `certifi`, `charset_normalizer`/`chardet`, `idna`). | +| **[`ARCHITECTURE.md`](ARCHITECTURE.md)** | End-to-end architecture diagram, separation between user-facing `Session`/`Request` models and transport `HTTPAdapter`/`urllib3` layer. | +| **[`DIRECTORY_MAP.md`](DIRECTORY_MAP.md)** | Directory tree annotated with responsibilities, entry points, and test harnesses. | +| **[`COMPONENT_CATALOG.md`](COMPONENT_CATALOG.md)** | Component directory indexing `Session`, `PreparedRequest`, `Response`, `HTTPAdapter`, `AuthBase`, and utility helpers. | +| **[`DEPENDENCY_MAP.md`](DEPENDENCY_MAP.md)** | Internal coupling and blast radius matrix; downstream consumer impact. | +| **[`DATA_FLOW.md`](DATA_FLOW.md)** | Request lifecycle state machine: un-prepared `Request` ➔ `PreparedRequest` ➔ `send()` ➔ raw socket response ➔ `Response`. | +| **[`API_AND_CONTRACTS.md`](API_AND_CONTRACTS.md)** | Public top-level API contracts (`requests.api`), session contracts, and adapter mount interfaces. | +| **[`DESIGN_PRINCIPLES.md`](DESIGN_PRINCIPLES.md)** | Human-centric design, backwards compatibility rules, error hierarchy conventions, and reference implementations. | +| **[`TEST_STRATEGY.md`](TEST_STRATEGY.md)** | Test runner harness, test server patterns (`tests/testserver/`), and recommended test commands for regression prevention. | +| **[`TEST_COVERAGE_GAPS.md`](TEST_COVERAGE_GAPS.md)** | Behavioral coverage gaps identified by static inspection (e.g. streaming teardown edge cases, custom proxy auth permutations). | +| **[`KNOWN_ISSUES.md`](KNOWN_ISSUES.md)** | Evidence-backed audit findings, historical architectural invariants, and gotchas. | +| **[`FEATURE_IMPLEMENTATION_GUIDE.md`](FEATURE_IMPLEMENTATION_GUIDE.md)** | Step-by-step recipe for safely extending `requests` without breaking public API invariants. | + +--- + +## 🤖 Agent Guides + +- **[`agents/QUICK_START.md`](agents/QUICK_START.md)**: 60-second orientation for autonomous coding agents. +- **[`agents/TASK_ROUTING.md`](agents/TASK_ROUTING.md)**: Task-to-file lookup table (e.g. "Modifying cookie handling", "Adding default adapter headers", "Adding proxy authentication"). +- **[`agents/REVIEW_CHECKLIST.md`](agents/REVIEW_CHECKLIST.md)**: Pre-commit / PR sanity checks for changes to `requests`. + +--- + +## 🔍 Deep-Dive Specifications + +### Components (`components/`) +- [`components/sessions.md`](components/sessions.md): `requests.sessions.Session` state management, cookie merging, redirect loops, and adapter routing. +- [`components/adapters.md`](components/adapters.md): `requests.adapters.HTTPAdapter` connection pooling, proxy resolution, and urllib3 bridging. +- [`components/models.md`](components/models.md): `Request`, `PreparedRequest`, and `Response` representations, encoding detection, and hook triggers. +- [`components/auth.md`](components/auth.md): Authentication dispatch (`HTTPBasicAuth`, `HTTPDigestAuth`, and custom `AuthBase` implementations). + +### Workflows (`workflows/`) +- [`workflows/get-request-trace.md`](workflows/get-request-trace.md): Step-by-step trace of `requests.get()` from call-site to HTTPAdapter socket read and `Response` return. + +--- + +## 📊 Manifest + +- **[`manifests/analysis_state.json`](manifests/analysis_state.json)**: Schema v2 manifest tracking analyzed commit hash, claim verification statuses, and file-dependency graph. diff --git a/examples/requests/.codeintel/KNOWN_ISSUES.md b/examples/requests/.codeintel/KNOWN_ISSUES.md new file mode 100644 index 0000000..ed7fb92 --- /dev/null +++ b/examples/requests/.codeintel/KNOWN_ISSUES.md @@ -0,0 +1,32 @@ +# Known Issues & Architectural Findings: `psf/requests` + +### [Finding REQ-001] Synchronous Blocking I/O Prevents Native Concurrency +- **Category:** Architecture / Performance +- **Severity:** Informational / Design Invariant +- **Status:** Known Limitation / Working as Designed +- **Confidence:** `VERIFIED` +- **Location:** Throughout `src/requests/` +- **Description:** Requests was deliberately constructed before standard Python `asyncio` maturation. It uses synchronous blocking calls (`socket.recv`) via `urllib3`. +- **Remediation / Workaround:** Applications needing concurrent non-blocking I/O should use thread pools (`concurrent.futures.ThreadPoolExecutor`) or modern async libraries like `httpx` or `aiohttp`. + +--- + +### [Finding REQ-002] Eager `response.content` Caches Entire Payload in Memory +- **Category:** Memory / Performance Risk +- **Severity:** Medium +- **Status:** Documented Behavior +- **Confidence:** `VERIFIED` +- **Location:** `src/requests/models.py:880-910` +- **Description:** Accessing `response.content` or `response.text` reads the entire socket stream into memory and saves it to `self._content`. Downloading multi-gigabyte files without `stream=True` and manual chunk consumption causes high memory spikes or out-of-memory crashes. +- **Remediation:** Always specify `stream=True` and iterate via `response.iter_content(chunk_size=...)` when transferring large files. + +--- + +### [Finding REQ-003] Ephemeral `requests.get()` Bypasses Connection Pool Reuse +- **Category:** Operational / Performance +- **Severity:** Low +- **Status:** Documented Behavior +- **Confidence:** `VERIFIED` +- **Location:** `src/requests/api.py:14-30` +- **Description:** Top-level functions like `requests.get(...)` create a new `Session()` inside a context manager for each call and immediately tear it down on return. This closes the underlying connection pool and eliminates TCP/TLS handshake reuse benefits across successive calls to the same host. +- **Remediation:** Reuse a `requests.Session()` instance across multiple requests to identical origins. diff --git a/examples/requests/.codeintel/PROJECT_PROFILE.md b/examples/requests/.codeintel/PROJECT_PROFILE.md new file mode 100644 index 0000000..1862624 --- /dev/null +++ b/examples/requests/.codeintel/PROJECT_PROFILE.md @@ -0,0 +1,23 @@ +# Project Profile: `psf/requests` + +## Domain Overview +`requests` is the standard Python synchronous HTTP library, designed around usability ("HTTP for Humans™"). It wraps low-level socket and connection-pool management in `urllib3` with a clean, pythonic object-oriented interface. + +## Repository Vital Statistics +- **Analyzed Release:** `v2.32.3` +- **Analyzed Commit:** `0e322af87745eff34caffe4df68456ebc20d9068` +- **Primary Language:** Python 3.8+ (supports 3.8 - 3.13) +- **Source Root:** `src/requests/` +- **Test Suite:** `tests/` (pytest-based, featuring custom threaded local HTTP test server in `tests/testserver/`) +- **Package Management & Build:** `setuptools`, `pyproject.toml`, `setup.py` + +## Core Dependencies +1. **`urllib3` (`>=1.21.1, <3`)**: The underlying transport engine managing connection pools, HTTP/HTTPS connection logic, SSL contexts, and low-level proxy tunneling. +2. **`certifi` (`>=2017.4.17`)**: Curated Mozilla root CA bundle for TLS certificate validation. +3. **`charset_normalizer` (`>=2, <4`)** or fallback **`chardet` (`>=3.0.2, <6`)**: Automatic character set detection for HTTP text responses (`response.text`). +4. **`idna` (`>=2.5, <4`)**: Internationalized Domain Names in Applications (RFC 5891) hostname encoding. + +## Primary Invariants +1. **Synchronous Execution:** All network I/O in core `requests` is blocking. No `async`/`await` primitives exist in core. +2. **Strict Backwards Compatibility:** Public API signatures (`requests.get`, `Session.request`, `PreparedRequest`, `Response`) remain stable across minor versions. +3. **Prepared State Separation:** Network mutations only occur on `PreparedRequest` objects, never on un-prepared `Request` objects. diff --git a/examples/requests/.codeintel/TEST_COVERAGE_GAPS.md b/examples/requests/.codeintel/TEST_COVERAGE_GAPS.md new file mode 100644 index 0000000..a049cd2 --- /dev/null +++ b/examples/requests/.codeintel/TEST_COVERAGE_GAPS.md @@ -0,0 +1,20 @@ +# Test Coverage Gaps: `psf/requests` + +> **Note:** The following findings are based on static code inspection of the test suite and source paths in `v2.32.3`. + +## 1. Streaming Teardown and Unconsumed Socket Edge Cases +- **Affected Path:** `src/requests/models.py:750-860` (`Response.iter_content`, `Response.close`) +- **Inspection Finding:** While `Response.close()` explicitly calls `self.raw.close()`, integration tests rarely verify connection pool health when an exception is raised mid-stream during an `iter_content(decode_unicode=True)` generator execution. +- **Risk:** Unconsumed raw sockets may remain checked out of the `urllib3` connection pool if caller fails to wrap the stream in a context manager, potentially causing connection pool starvation. +- **Confidence:** `INFERRED` + +## 2. Dynamic Complex Proxy Configuration Combinations +- **Affected Path:** `src/requests/adapters.py:220-270` (`HTTPAdapter.proxy_headers`) +- **Inspection Finding:** Test coverage for socks proxies with mutual TLS authentication (mTLS) combined with HTTP fallback proxies is sparse due to lack of local mock infrastructure for multi-hop proxy tunnels. +- **Risk:** Regressions in urllib3 proxy tunneling upgrades may go undetected in pure unit tests. +- **Confidence:** `OBSERVED` + +## 3. Redirect Strip Auth on Scheme Downgrade or Port Change +- **Affected Path:** `src/requests/sessions.py:220-300` (`SessionRedirectMixin.should_strip_auth`) +- **Inspection Finding:** Test cases cover stripping `Authorization` headers when redirecting from HTTPS to HTTP or cross-domain, but coverage for non-standard port shifts on the same hostname (e.g. `example.com:8443` -> `example.com:443`) is minimal. +- **Confidence:** `VERIFIED` diff --git a/examples/requests/.codeintel/TEST_STRATEGY.md b/examples/requests/.codeintel/TEST_STRATEGY.md new file mode 100644 index 0000000..bfb6ad8 --- /dev/null +++ b/examples/requests/.codeintel/TEST_STRATEGY.md @@ -0,0 +1,28 @@ +# Test Strategy: `psf/requests` + +## Test Suite Architecture + +The Requests test suite (`tests/`) uses `pytest` and is divided into: +1. **Unit Tests (`tests/test_structures.py`, `tests/test_utils.py`, `tests/test_hooks.py`)**: Fast, memory-only tests verifying header casing, parameter quoting, and hook invocation. +2. **Integration Tests (`tests/test_requests.py`, `tests/test_adapters.py`)**: Tests that exercise request execution, redirect chains, cookie persistence, and proxy resolution. +3. **Local Threaded Test Server (`tests/testserver/`)**: An in-process HTTP server enabling network integration testing without calling out to external web services. + +## Recommended Test Commands + +```bash +# Run entire test suite +pytest + +# Run fast unit tests only (no socket I/O) +pytest tests/test_structures.py tests/test_utils.py tests/test_hooks.py + +# Run adapter & connection pooling tests +pytest tests/test_adapters.py + +# Run low-level network and redirect tests +pytest tests/test_lowlevel.py tests/test_requests.py -k "redirect" +``` + +## Test Invariants +- Integration tests must not depend on internet access; tests make use of `tests/testserver/` or `httpbin` mocks. +- SSL tests rely on locally generated fixture certificates located in `tests/certs/`. diff --git a/examples/requests/.codeintel/agents/QUICK_START.md b/examples/requests/.codeintel/agents/QUICK_START.md new file mode 100644 index 0000000..0e578ad --- /dev/null +++ b/examples/requests/.codeintel/agents/QUICK_START.md @@ -0,0 +1,31 @@ +# Agent Quick Start: `psf/requests` + +Welcome, AI Agent! Use this quick-start guide to navigate `psf/requests` without re-reading the entire codebase. + +--- + +## ⚡ 60-Second Orientation +- **Core Orchestrator:** `src/requests/sessions.py` (`Session`). +- **Data Models:** `src/requests/models.py` (`Request`, `PreparedRequest`, `Response`). +- **Network Transport:** `src/requests/adapters.py` (`HTTPAdapter`). +- **Top-Level Entry Point:** `src/requests/api.py`. +- **Underlying Engine:** `urllib3` (manages connections and pools). + +--- + +## 🧭 Workflow Handoff Table + +| When You Are Doing... | Start By Reading... | Key Rule to Remember | +| :--- | :--- | :--- | +| **Planning a feature** | `.codeintel/FEATURE_IMPLEMENTATION_GUIDE.md` | Any new parameter must be threaded through `api.py` ➔ `sessions.py` ➔ `models.py`/`adapters.py`. | +| **Debugging network errors** | `src/requests/adapters.py:510-600` | All `urllib3` errors must be caught and converted to `requests.exceptions`. | +| **Working on Cookies** | `src/requests/cookies.py` | Requests uses `RequestsCookieJar`, which subclasses Python's standard `http.cookiejar.CookieJar`. | +| **Reviewing PRs / Changes** | `.codeintel/agents/REVIEW_CHECKLIST.md` | Ensure backwards compatibility and verify no network requests leak in unit tests. | + +--- + +## 🛠️ Verification Command +```bash +pytest +``` +Run tests locally before concluding any implementation task. diff --git a/examples/requests/.codeintel/agents/REVIEW_CHECKLIST.md b/examples/requests/.codeintel/agents/REVIEW_CHECKLIST.md new file mode 100644 index 0000000..a43813b --- /dev/null +++ b/examples/requests/.codeintel/agents/REVIEW_CHECKLIST.md @@ -0,0 +1,36 @@ +# Code Review Checklist: `psf/requests` + +Before accepting, committing, or proposing changes to `requests`, verify every item below: + +--- + +## 1. Backwards Compatibility +- [ ] No public method signatures in `requests.api`, `Session`, `PreparedRequest`, or `Response` have had arguments removed or reordered. +- [ ] All new kwargs have safe default values (`None` or `False`). +- [ ] Existing exceptions remain subclasses of `requests.exceptions.RequestException`. + +--- + +## 2. Transport Separation & Leak Prevention +- [ ] No `urllib3` internal objects or exceptions leak directly to caller without being wrapped or converted. +- [ ] In `HTTPAdapter.send()`, every new socket/urllib3 error condition maps to a corresponding `requests.exceptions.*` class. +- [ ] `PreparedRequest` does not initiate socket connections or invoke transport adapter methods. + +--- + +## 3. Streaming & Resource Cleanup +- [ ] If changing `Response` or `HTTPAdapter.send`, verify that unstreamed requests automatically release connections back to the pool. +- [ ] Streaming requests (`stream=True`) leave the raw stream accessible until `.close()` or full generator exhaustion. + +--- + +## 4. Security Invariants +- [ ] Cross-domain redirects strip sensitive headers (`Authorization`, `Cookie`) as defined in `SessionRedirectMixin.should_strip_auth`. +- [ ] HTTPS to HTTP downgrades strip credentials. +- [ ] Default `verify=True` remains intact across all entry points. + +--- + +## 5. Verification +- [ ] Pytest suite executes and passes 100% offline. +- [ ] Regression test added under `tests/` reproducing the exact bug or exercising the new feature. diff --git a/examples/requests/.codeintel/agents/TASK_ROUTING.md b/examples/requests/.codeintel/agents/TASK_ROUTING.md new file mode 100644 index 0000000..1bc5573 --- /dev/null +++ b/examples/requests/.codeintel/agents/TASK_ROUTING.md @@ -0,0 +1,15 @@ +# Task Routing Guide: `psf/requests` + +Use this routing table to jump directly to the exact file and lines responsible for a behavior: + +| Developer Task | Relevant Source Files | Reference Lines | +| :--- | :--- | :--- | +| **Change how URLs or query parameters are parsed** | `src/requests/models.py` | `PreparedRequest.prepare_url` (~lines 380-450) | +| **Add or alter default request headers** | `src/requests/utils.py` | `default_headers`, `default_user_agent` | +| **Fix cookie jar merging between Session and Request** | `src/requests/sessions.py`
`src/requests/cookies.py` | `Session.request` (~lines 510-540)
`merge_cookies` | +| **Modify HTTP redirect behavior or header stripping** | `src/requests/sessions.py` | `SessionRedirectMixin.resolve_redirects`
`should_strip_auth` | +| **Update SSL certificate verification or CA paths** | `src/requests/adapters.py` | `HTTPAdapter.cert_verify` | +| **Adjust connection pooling defaults (size, block)** | `src/requests/adapters.py` | `HTTPAdapter.__init__`, `init_poolmanager` | +| **Add or change authentication scheme** | `src/requests/auth.py` | Subclass `AuthBase` | +| **Translate a new urllib3 error** | `src/requests/adapters.py` | `HTTPAdapter.send` exception handler block | +| **Change JSON deserialization behavior in Response** | `src/requests/models.py` | `Response.json` (~lines 950-1000) | diff --git a/examples/requests/.codeintel/components/adapters.md b/examples/requests/.codeintel/components/adapters.md new file mode 100644 index 0000000..aac3d5b --- /dev/null +++ b/examples/requests/.codeintel/components/adapters.md @@ -0,0 +1,20 @@ +# Component: `HTTPAdapter` (`requests.adapters`) + +## Overview +`HTTPAdapter` is the built-in transport adapter that connects `requests` to `urllib3`. It manages the creation and pooling of socket connections through `urllib3.PoolManager` and handles proxy tunneling through `urllib3.ProxyManager`. + +## Source Location +`src/requests/adapters.py:54` + +## Key Responsibilities +1. **Connection Pool Management:** Instantiates and configures `urllib3.PoolManager` with pool sizing (`pool_connections`, `pool_maxsize`), max retries, and SSL certificate context. +2. **Proxy Resolution:** Detects environment proxies (`get_environ_proxies`) and mounts `ProxyManager` instances as needed. +3. **PreparedRequest Execution:** Takes a `PreparedRequest`, extracts target connection host/port, checks out a connection from `PoolManager`, and issues the low-level `urlopen()` call. +4. **Exception Translation:** Catches all `urllib3` errors (e.g. `MaxRetryError`, `SSLError`, `TimeoutError`, `ProtocolError`) and re-raises them as unified `requests.exceptions.*` errors. +5. **Response Packaging:** Takes the `urllib3.response.HTTPResponse` and wraps it into a `requests.models.Response` instance via `build_response()`. + +## Customization Patterns +Subclassing `HTTPAdapter` is the official mechanism in Requests for: +- Implementing custom retry logic using `urllib3.util.Retry`. +- Adjusting socket timeout defaults or TCP keepalive options. +- Binding connections to specific network interfaces or SSL ciphers. diff --git a/examples/requests/.codeintel/components/auth.md b/examples/requests/.codeintel/components/auth.md new file mode 100644 index 0000000..c684175 --- /dev/null +++ b/examples/requests/.codeintel/components/auth.md @@ -0,0 +1,24 @@ +# Component: `auth` (`requests.auth`) + +## Overview +The `requests.auth` module defines the authentication framework for Requests. It centers around `AuthBase`, a callable interface invoked during `PreparedRequest` construction. + +## Source Location +`src/requests/auth.py` + +## Core Classes + +### 1. `AuthBase` (line 62) +- Base class for all authentication handlers. +- **Contract:** Subclasses implement `__call__(self, r)` where `r` is a `PreparedRequest` instance. The handler modifies `r.headers` or other attributes and returns `r`. + +### 2. `HTTPBasicAuth` (line 72) +- Standard HTTP Basic Authentication. +- Computes base64-encoded `'Basic ' + b64encode(username:password)` and attaches it to `r.headers['Authorization']`. + +### 3. `HTTPDigestAuth` (line 104) +- Challenge-response HTTP Digest Authentication (RFC 2617). +- Stateful: catches initial 401 response in a response hook, parses `WWW-Authenticate: Digest` challenge headers, generates the digest hash, and replays the request with the `Authorization: Digest ...` header. + +## Invariant +- Authentication handlers operate directly on `PreparedRequest` before transport dispatch, or register response hooks to handle authentication challenges (like Digest or OAuth refreshes). diff --git a/examples/requests/.codeintel/components/models.md b/examples/requests/.codeintel/components/models.md new file mode 100644 index 0000000..124901a --- /dev/null +++ b/examples/requests/.codeintel/components/models.md @@ -0,0 +1,37 @@ +# Component: `models` (`Request`, `PreparedRequest`, `Response`) + +## Overview +The `requests.models` module defines the data transfer objects passed across the library layers. + +## Source Location +`src/requests/models.py` + +## Core Classes + +### 1. `Request` (line 204) +- **Role:** Pure container holding user-specified configuration (method, url, headers, files, data, params, auth, cookies, hooks, json). +- **Invariants:** Not transmitted over the wire. Must be converted into a `PreparedRequest` via `req.prepare()` or `session.prepare_request(req)`. + +### 2. `PreparedRequest` (line 277) +- **Role:** Complete, mutable wire representation of an HTTP request. +- **Attributes:** + - `method`: HTTP verb (e.g. `'GET'`, `'POST'`). + - `url`: Canonical URL string with query parameters encoded and host IDNA-encoded. + - `headers`: `CaseInsensitiveDict` of HTTP headers. + - `body`: Serialized byte string or file-like stream. + - `hooks`: Event hook callbacks. +- **Methods:** `prepare(method, url, headers, files, data, params, auth, cookies, hooks, json)`. + +### 3. `Response` (line 601) +- **Role:** Represents the received HTTP response. +- **Key Attributes:** + - `status_code`: Integer HTTP status code. + - `headers`: Case-insensitive response headers. + - `raw`: Underlying `urllib3.response.HTTPResponse` object. + - `history`: List of prior `Response` objects generated during redirects. + - `encoding`: Guessed or declared text encoding (uses `charset_normalizer` or `chardet`). +- **Key Properties / Methods:** + - `content`: Returns entire response body as bytes, caching it in `_content`. + - `text`: Decodes `content` using `encoding`. + - `json(**kwargs)`: Parses `text` using standard `json.loads`. + - `raise_for_status()`: Raises `HTTPError` if `400 <= status_code < 600`. diff --git a/examples/requests/.codeintel/components/sessions.md b/examples/requests/.codeintel/components/sessions.md new file mode 100644 index 0000000..7590640 --- /dev/null +++ b/examples/requests/.codeintel/components/sessions.md @@ -0,0 +1,24 @@ +# Component: `Session` (`requests.sessions`) + +## Overview +`Session` is the stateful HTTP client object in `requests`. It preserves cookie jars across requests, holds persistent configuration (authentication, default headers, proxies, SSL verification settings), and manages transport adapter routing via mounted URL prefixes. + +## Source Location +`src/requests/sessions.py:348` + +## Key Responsibilities +1. **Cookie Jar Maintenance:** Merges incoming `Set-Cookie` response headers into `self.cookies` (`RequestsCookieJar`) and attaches them to subsequent outgoing requests. +2. **Adapter Routing:** Maintains `self.adapters` mapping prefix strings (e.g., `'https://'`, `'http://'`) to `BaseAdapter` implementations. +3. **Redirect Loop Execution:** Implements `SessionRedirectMixin.resolve_redirects` to handle HTTP 301, 302, 303, 307, and 308 response codes up to `self.max_redirects` (default 30). +4. **Auth Persistence:** Holds `self.auth` which is applied to any request where caller does not specify per-call `auth`. + +## Critical Public Methods +- `request(method, url, ...)`: Constructs `Request`, passes to `prepare_request()`, sends via `send()`. +- `prepare_request(request)`: Merges session settings with the given `Request` and returns a `PreparedRequest`. +- `send(request, **kwargs)`: Dispatches `PreparedRequest` to the matching adapter, processes redirects, runs response hooks. +- `mount(prefix, adapter)`: Registers a transport adapter for a given URL scheme or domain prefix. +- `close()`: Closes all mounted adapters, flushing and closing underlying connection pools. + +## Invariants +- A `Session` is not thread-safe for simultaneous mutating requests using shared cookies, though underlying `urllib3` connection pools are thread-safe. +- Must be used as a context manager (`with requests.Session() as s:`) or explicitly closed to avoid lingering open sockets. diff --git a/examples/requests/.codeintel/manifests/analysis_state.json b/examples/requests/.codeintel/manifests/analysis_state.json new file mode 100644 index 0000000..3595a85 --- /dev/null +++ b/examples/requests/.codeintel/manifests/analysis_state.json @@ -0,0 +1,93 @@ +{ + "schema_version": 2, + "repository": { + "name": "psf/requests", + "analyzed_commit": "0e322af87745eff34caffe4df68456ebc20d9068", + "analyzed_branch": "v2.32.3", + "working_tree_clean": true, + "includes_uncommitted_changes": false + }, + "analysis": { + "generated_at": "2026-10-31T09:31:00Z", + "mode": "deep_analysis", + "status": "complete", + "unresolved_items": [] + }, + "artifacts": { + "INDEX.md": { + "depends_on": ["src/requests/__init__.py", "src/requests/api.py"], + "status": "current" + }, + "PROJECT_PROFILE.md": { + "depends_on": ["setup.py", "pyproject.toml"], + "status": "current" + }, + "ARCHITECTURE.md": { + "depends_on": ["src/requests/api.py", "src/requests/sessions.py", "src/requests/adapters.py", "src/requests/models.py"], + "status": "current" + }, + "DIRECTORY_MAP.md": { + "depends_on": ["src/requests/"], + "status": "current" + }, + "COMPONENT_CATALOG.md": { + "depends_on": ["src/requests/sessions.py", "src/requests/adapters.py", "src/requests/models.py"], + "status": "current" + }, + "DEPENDENCY_MAP.md": { + "depends_on": ["src/requests/api.py", "src/requests/sessions.py", "src/requests/adapters.py"], + "status": "current" + }, + "DATA_FLOW.md": { + "depends_on": ["src/requests/api.py", "src/requests/sessions.py", "src/requests/models.py", "src/requests/adapters.py"], + "status": "current" + }, + "API_AND_CONTRACTS.md": { + "depends_on": ["src/requests/api.py", "src/requests/adapters.py"], + "status": "current" + }, + "DESIGN_PRINCIPLES.md": { + "depends_on": ["src/requests/adapters.py", "src/requests/auth.py"], + "status": "current" + }, + "TEST_STRATEGY.md": { + "depends_on": ["tests/conftest.py", "tests/test_requests.py"], + "status": "current" + }, + "TEST_COVERAGE_GAPS.md": { + "depends_on": ["src/requests/models.py", "src/requests/adapters.py", "src/requests/sessions.py"], + "status": "current" + }, + "KNOWN_ISSUES.md": { + "depends_on": ["src/requests/models.py", "src/requests/api.py"], + "status": "current" + }, + "FEATURE_IMPLEMENTATION_GUIDE.md": { + "depends_on": ["src/requests/api.py", "src/requests/sessions.py", "src/requests/adapters.py"], + "status": "current" + } + }, + "claims": [ + { + "claim_id": "CLM-REQ-001", + "statement": "Calling requests.get creates and disposes an ephemeral Session instance", + "status": "VERIFIED", + "evidence_path": "src/requests/api.py", + "evidence_lines": "14-64" + }, + { + "claim_id": "CLM-REQ-002", + "statement": "HTTPAdapter manages urllib3.PoolManager connection pooling and catches urllib3 exceptions", + "status": "VERIFIED", + "evidence_path": "src/requests/adapters.py", + "evidence_lines": "54-600" + }, + { + "claim_id": "CLM-REQ-003", + "statement": "SessionRedirectMixin strips sensitive Authorization headers on domain cross-origin redirects", + "status": "VERIFIED", + "evidence_path": "src/requests/sessions.py", + "evidence_lines": "220-300" + } + ] +} diff --git a/examples/requests/.codeintel/workflows/get-request-trace.md b/examples/requests/.codeintel/workflows/get-request-trace.md new file mode 100644 index 0000000..c4ba0d6 --- /dev/null +++ b/examples/requests/.codeintel/workflows/get-request-trace.md @@ -0,0 +1,78 @@ +# Workflow Trace: `requests.get()` End-to-End + +This document traces the complete execution path of a call to `requests.get('https://example.com/api', params={'q': 'python'}, headers={'User-Agent': 'my-app'})`. + +--- + +## Step 1: Top-Level Entry Point (`src/requests/api.py:64`) +```python +def get(url, params=None, **kwargs): + return request("get", url, params=params, **kwargs) +``` +- Calls `requests.api.request('get', url, params=params, **kwargs)`. +- Instantiates an ephemeral `Session` inside a context manager: + ```python + with sessions.Session() as session: + return session.request(method=method, url=url, **kwargs) + ``` + +--- + +## Step 2: Session Orchestration (`src/requests/sessions.py:510`) +1. `Session.request(...)` constructs an un-prepared `Request`: + ```python + req = Request( + method=method.upper(), + url=url, + headers=headers, + files=files, + data=data or {}, + json=json, + params=params or {}, + auth=auth, + cookies=cookies, + hooks=hooks, + ) + ``` +2. Passes `req` to `session.prepare_request(req)`. + - Merges session headers, session cookies, and session authentication. + - Instantiates `p = PreparedRequest()`. + - Calls `p.prepare(...)`. + - `prepare_url`: Encodes query params into `https://example.com/api?q=python`. + - `prepare_headers`: Normalizes into `CaseInsensitiveDict`. + - `prepare_cookies`: Encodes cookies into header string. + - `prepare_auth`: Invokes auth callback. +3. Passes `p` to `session.send(prep, **send_kwargs)`. + +--- + +## Step 3: Adapter Lookup & Transport Dispatch (`src/requests/sessions.py:734`) +1. `session.get_adapter(url=prep.url)` scans `self.adapters` keys by prefix length. +2. Matches `'https://'` prefix and returns the mounted `HTTPAdapter` instance. +3. Calls `adapter.send(prep, stream=False, timeout=None, verify=True, cert=None, proxies=...)`. + +--- + +## Step 4: Urllib3 Pool Connection & Socket I/O (`src/requests/adapters.py:440`) +1. `adapter.get_connection(prep.url, proxies)` checks out a `urllib3.connectionpool.HTTPSConnectionPool` for `example.com:443`. +2. Formats headers and path. +3. Calls `conn.urlopen(method=prep.method, url=url, body=prep.body, headers=prep.headers, ...)` on `urllib3`. +4. `urllib3` establishes TLS handshake (verifying certificates using `certifi` CA bundle), sends HTTP headers and body over socket, and reads HTTP response status and headers. +5. `urllib3` returns a `urllib3.response.HTTPResponse` object. + +--- + +## Step 5: Response Construction (`src/requests/adapters.py:600`) +1. `adapter.build_response(prep, raw_response)` creates a `requests.models.Response` object. +2. Sets `response.status_code`, `response.headers`, `response.raw = raw_response`, `response.url = prep.url`. +3. Attaches response cookies to `response.cookies`. + +--- + +## Step 6: Redirect Check & Session Cleanup (`src/requests/sessions.py:660`) +1. Checks `response.is_redirect` (status codes 301, 302, 303, 307, 308). + - If redirected and `allow_redirects=True`: calls `resolve_redirects()`, creating a generator that executes subsequent requests and appends intermediate responses to `response.history`. + - If not redirected: proceeds to hook execution. +2. Executes any registered response event hooks (`dispatch_hook('response', ...)`). +3. Closes ephemeral session context, which calls `Session.close()` (closing adapters). +4. Returns `response` to caller. diff --git a/examples/requests/README.md b/examples/requests/README.md new file mode 100644 index 0000000..63df4b8 --- /dev/null +++ b/examples/requests/README.md @@ -0,0 +1,22 @@ +# Example: `psf/requests` + +This directory demonstrates what **CodeIntel** creates inside an actual repository. + +When an AI coding agent runs the `codeintel` skill on a target repo, it builds and maintains an evidence-backed knowledge base inside a local `.codeintel/` directory. + +--- + +## 📂 Knowledge Base Location + +The full generated knowledge base for **`psf/requests` (`v2.32.3`)** is located in: + +👉 **[`examples/requests/.codeintel/`](.codeintel/)** + +### Key Documents to Explore +- **[`.codeintel/INDEX.md`](.codeintel/INDEX.md)**: Entry point & task router. +- **[`.codeintel/ARCHITECTURE.md`](.codeintel/ARCHITECTURE.md)**: Actual implemented architecture with Mermaid component diagrams. +- **[`.codeintel/workflows/get-request-trace.md`](.codeintel/workflows/get-request-trace.md)**: End-to-end trace of `requests.get()` across socket boundaries. +- **[`.codeintel/TEST_COVERAGE_GAPS.md`](.codeintel/TEST_COVERAGE_GAPS.md)**: Behavioral test gaps identified by static inspection. +- **[`.codeintel/KNOWN_ISSUES.md`](.codeintel/KNOWN_ISSUES.md)**: Evidence-backed findings with explicit confidence levels. +- **[`.codeintel/agents/QUICK_START.md`](.codeintel/agents/QUICK_START.md)**: 60-second orientation for autonomous agents. +- **[`.codeintel/manifests/analysis_state.json`](.codeintel/manifests/analysis_state.json)**: Schema v2 manifest with checkpoint hash and claim verification statuses.