Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 155 additions & 43 deletions README.md

Large diffs are not rendered by default.

Binary file added assets/social-preview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
48 changes: 48 additions & 0 deletions examples/requests/.codeintel/API_AND_CONTRACTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# API & Contracts: `psf/requests`

## Top-Level Public Interface (`requests.*`)

The public interface exposed at `requests.*` maps directly to `requests.api`:

```python
requests.request(method, url, **kwargs)
requests.get(url, params=None, **kwargs)
requests.post(url, data=None, json=None, **kwargs)
requests.put(url, data=None, **kwargs)
requests.patch(url, data=None, **kwargs)
requests.delete(url, **kwargs)
requests.head(url, **kwargs)
requests.options(url, **kwargs)
```

### Accepted `**kwargs` Parameters
- `params`: Dictionary, list of tuples or bytes to append to query string.
- `data`: Dictionary, list of tuples, bytes, or file-like object to send in the body.
- `json`: A JSON-serializable python object to send in the body.
- `headers`: Dictionary of HTTP Headers to send with the `Request`.
- `cookies`: Dict or CookieJar object to send with the `Request`.
- `files`: Dictionary of `'name': file-like-objects` (or `('filename', fileobj)`) for multipart encoding.
- `auth`: Auth tuple or callable object for HTTP authentication.
- `timeout`: Float describing timeout of the request in seconds, or a `(connect_timeout, read_timeout)` tuple.
- `allow_redirects`: Boolean. Enable/disable GET/OPTIONS/POST/etc. redirection handling.
- `proxies`: Dictionary mapping protocol to the URL of the proxy.
- `verify`: Boolean (whether to verify SSL cert) or string (path to CA bundle).
- `stream`: Boolean. If `False`, response content will be immediately downloaded.
- `cert`: String (path to SSL client cert file) or tuple `('cert', 'key')`.

---

## Adapter Contract (`requests.adapters.BaseAdapter`)

Any transport adapter mounted via `session.mount(prefix, adapter)` must adhere to:

```python
class BaseAdapter:
def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
"""Sends PreparedRequest object. Returns Response object."""
raise NotImplementedError

def close(self):
"""Cleans up adapter-specific resources."""
raise NotImplementedError
```
55 changes: 55 additions & 0 deletions examples/requests/.codeintel/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Architecture: `psf/requests`

Requests is organized into four distinct layers:

1. **Facade Layer (`requests.api` & `requests.__init__`)**: Convenience functions (`get`, `post`, `put`, `delete`, `head`, `options`, `patch`, `request`) that manage ephemeral `Session` instances.
2. **Session & Orchestration Layer (`requests.sessions`)**: Maintains cookies across requests, stores auth/header defaults, resolves redirects (`SessionRedirectMixin`), and routes requests to transport adapters based on URL prefixes.
3. **Data Model Layer (`requests.models`)**:
- `Request`: Raw request configuration supplied by caller.
- `PreparedRequest`: Immutable request payload (headers formatted, body encoded, URL canonicalized with query parameters, cookies attached) ready for socket dispatch.
- `Response`: Encapsulates status code, headers, cookies, raw byte streaming (`iter_content`), text decoding (`text`), and redirect history.
4. **Transport Layer (`requests.adapters` & `urllib3`)**:
- `BaseAdapter`: Abstract contract for transport handlers.
- `HTTPAdapter`: Concrete adapter that manages `urllib3.PoolManager` and `urllib3.ProxyManager`, translates `PreparedRequest` into `urllib3` calls, and packages the result into a `Response`.

---

## Architectural Workflow Diagram

```mermaid
flowchart TD
User([Caller / Application Code]) -->|requests.get / post| Facade[requests.api]
Facade -->|instantiates ephemeral| Session[requests.sessions.Session]
User -.->|or uses persistent| Session

Session -->|1. prepare_request| RequestModel[requests.models.Request]
RequestModel -->|prepare| PrepReq[requests.models.PreparedRequest]

Session -->|2. get_adapter for URL prefix| AdapterRegistry{Adapter Mount Map}
AdapterRegistry -->|http:// or https://| HTTPAdapter[requests.adapters.HTTPAdapter]

HTTPAdapter -->|3. send PreparedRequest| PoolMgr[urllib3.PoolManager / ProxyManager]
PoolMgr -->|4. socket I/O| RemoteServer[(Target HTTP/HTTPS Host)]
RemoteServer -->|5. raw urllib3 response| PoolMgr
PoolMgr -->|urllib3 HTTPResponse| HTTPAdapter

HTTPAdapter -->|6. build_response| ResponseModel[requests.models.Response]
ResponseModel -->|7. attach history / resolve redirects| Session
Session -->|8. return Response| User
```

---

## Key Architectural Decisions & Evidence

### 1. Ephemeral vs. Persistent Sessions
- **Evidence:** `src/requests/api.py:14-64`
- Calling `requests.get()` opens an ephemeral `with sessions.Session() as session:` block. Each top-level call builds and tears down a connection pool unless the caller explicitly uses `requests.Session()`.

### 2. URL Prefix Adapter Dispatch
- **Evidence:** `src/requests/sessions.py:734-754` (`Session.get_adapter`)
- Sessions store an ordered dictionary of adapters mounted to prefixes (`self.adapters = CaseInsensitiveDict()`). Standard setup mounts `https://` and `http://` to default `HTTPAdapter` instances. Custom adapters can be mounted for custom schemes or specific domains.

### 3. Preparation Pipeline
- **Evidence:** `src/requests/models.py:277-505` (`PreparedRequest.prepare`)
- Preparation occurs sequentially: `prepare_method`, `prepare_url`, `prepare_headers`, `prepare_cookies`, `prepare_body`, `prepare_auth`, `prepare_hooks`. Each step mutates only the `PreparedRequest` instance.
29 changes: 29 additions & 0 deletions examples/requests/.codeintel/COMPONENT_CATALOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Component Catalog: `psf/requests`

### 1. `Session` (`requests.sessions.Session`)
- **Location:** `src/requests/sessions.py:348`
- **Role:** Central orchestrator for request execution and client-side state (cookies, headers, authentication, adapter registry).
- **Key Methods:** `request()`, `send()`, `mount()`, `prepare_request()`, `resolve_redirects()`.
- **Contracts:** Inherits from `SessionRedirectMixin`. Must be closed via `.close()` or used as a context manager (`with Session() as s:`) to properly dispose underlying connection pools.

### 2. `PreparedRequest` (`requests.models.PreparedRequest`)
- **Location:** `src/requests/models.py:277`
- **Role:** Holds the fully evaluated request payload. Prepared requests contain canonicalized headers, serialized body streams, and full URLs.
- **Key Methods:** `prepare()`, `prepare_url()`, `prepare_headers()`, `prepare_body()`, `prepare_auth()`, `copy()`.
- **Invariants:** Does not execute network I/O. Safe to clone and inspect before execution.

### 3. `Response` (`requests.models.Response`)
- **Location:** `src/requests/models.py:601`
- **Role:** Encapsulates server HTTP response metadata and body content.
- **Key Attributes / Methods:** `status_code`, `headers`, `content` (raw bytes), `text` (decoded str), `json()`, `iter_content()`, `raise_for_status()`.
- **Contracts:** `response.raw` holds the raw `urllib3.response.HTTPResponse` object. Accessing `.content` caches all bytes in memory.

### 4. `HTTPAdapter` (`requests.adapters.HTTPAdapter`)
- **Location:** `src/requests/adapters.py:54`
- **Role:** Bridges `requests` with `urllib3.PoolManager`. Manages socket connection pools, TLS configurations (`cert`, `verify`), and translates `urllib3` exceptions into `requests.exceptions`.
- **Key Methods:** `send()`, `build_response()`, `get_connection()`, `close()`.
- **Invariants:** Owns connection pool lifecycles. Can be subclassed to implement custom retry logic, circuit breakers, or custom transports.

### 5. `CaseInsensitiveDict` (`requests.structures.CaseInsensitiveDict`)
- **Location:** `src/requests/structures.py:13`
- **Role:** Specialized mapping structure for HTTP headers where lookup is case-insensitive, while preserving the casing of original keys when iterated over.
53 changes: 53 additions & 0 deletions examples/requests/.codeintel/DATA_FLOW.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Data Flow & Lifecycle: `psf/requests`

## The Request Lifecycle

```text
1. Caller Invocation
requests.get(url, params=..., headers=...)
│
▼
2. Ephemeral Session Creation (in requests.api)
session = Session()
session.request('GET', url, ...)
│
▼
3. Request Construction & Preparation
req = Request(method='GET', url=url, headers=headers, files=..., data=..., json=..., params=...)
prep = session.prepare_request(req)
[prep.prepare_url: parameters encoded, idna resolved]
[prep.prepare_headers: CaseInsensitiveDict initialized]
[prep.prepare_auth: HTTPBasicAuth or custom auth applied]
[prep.prepare_cookies: Session cookies merged with Request cookies]
│
▼
4. Adapter Resolution
adapter = session.get_adapter(url=prep.url) # matches "http://" or "https://" prefix
│
▼
5. Adapter Execution
resp = adapter.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
[urllib3 connection pool checked out]
[Low-level urllib3.response.HTTPResponse received]
[requests.models.Response constructed via adapter.build_response()]
│
▼
6. Redirect Resolution (if resp.is_redirect and allow_redirects=True)
generator = session.resolve_redirects(resp, prep, stream=stream, ...)
[Loops through 301/302/303/307/308 responses up to max_redirects (default 30)]
[History appended to final response.history]
│
▼
7. Hook Dispatch & Return
resp = dispatch_hook('response', prep.hooks, resp)
return resp
```

## Response Body Consumption Flow

1. **Eager Mode (`stream=False`, default):**
- `adapter.build_response()` does not read body immediately, but `Session.send()` does not close it.
- Calling `response.content` invokes `response._content = b"".join(response.iter_content(CONTENT_CHUNK_SIZE))`, eagerly loading all bytes and releasing the connection back to the `urllib3` pool.
2. **Streaming Mode (`stream=True`):**
- Socket connection stays open and attached to `response.raw`.
- Caller must iterate via `response.iter_content()` or `response.iter_lines()`, or explicitly call `response.close()`.
31 changes: 31 additions & 0 deletions examples/requests/.codeintel/DEPENDENCY_MAP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Dependency Map & Blast Radius: `psf/requests`

## Internal Coupling Graph

```text
requests.api
└── requests.sessions.Session
├── requests.models.Request
│ └── requests.models.PreparedRequest
├── requests.adapters.HTTPAdapter (implements BaseAdapter)
│ └── urllib3.PoolManager / ProxyManager
├── requests.cookies.RequestsCookieJar
└── requests.auth (AuthBase, HTTPBasicAuth)
```

## Blast Radius Analysis

### High Blast Radius Components
1. **`requests.models.PreparedRequest`**
- **Impact Level:** 🚨 Critical
- **Reason:** Every adapter, hook, and authentication handler operates on `PreparedRequest`. Any modification to its internal attributes (`.url`, `.headers`, `.body`, `.hooks`) affects all transport adapters and session pipelines.
2. **`requests.sessions.Session.send`**
- **Impact Level:** 🚨 Critical
- **Reason:** Core dispatch pipeline responsible for resolving adapter targets, attaching cookies, and executing redirect loops. Changes here affect every HTTP request in the library.
3. **`requests.adapters.HTTPAdapter.send`**
- **Impact Level:** 🚨 Critical
- **Reason:** Direct bridge to `urllib3`. Handles exception translation, timeout mapping, proxy configuration, and certificate validation.

### Low Blast Radius Components
1. **`requests.status_codes`**: Static lookup dict; modifying entries affects only status symbol resolution (`codes.ok`).
2. **`requests.help`**: Informational module printing environment diagnostics; has zero runtime side effects on HTTP operations.
32 changes: 32 additions & 0 deletions examples/requests/.codeintel/DESIGN_PRINCIPLES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Design Principles & Reference Implementations: `psf/requests`

## Core Design Principles

1. **Explicit Ergonomics over Obscure Configuration:** The API defaults to what 99% of developers need (automatic redirect following on GET, JSON decoding helper, connection pooling, cert verification enabled by default).
2. **Defensive Translation of Underlying Exceptions:** Never leak `urllib3` exceptions directly to the user. Every `urllib3` error must be caught in `HTTPAdapter.send` and re-raised as a corresponding `requests.exceptions.RequestException` subclass.
3. **Pluggable Architecture via Adapters:** Transport-level variations (retries, UNIX domain sockets, mock backends) are extended via `HTTPAdapter` subclasses mounted to URL prefixes, keeping the `Session` orchestrator clean.

---

## Canonical Reference Patterns

### Pattern 1: Translation of Transport Exceptions
- **Reference:** `src/requests/adapters.py:510-600`
- **Why it is a good reference:** Demonstrates the canonical pattern for catching low-level socket/HTTP client errors and converting them into high-level user-friendly exceptions without losing traceback context.

### Pattern 2: Custom Authentication via `AuthBase`
- **Reference:** `src/requests/auth.py:72-102` (`HTTPBasicAuth`)
- **Implementation Guide:**
```python
from requests.auth import AuthBase

class CustomTokenAuth(AuthBase):
def __init__(self, token):
self.token = token

def __call__(self, r):
# Mutate PreparedRequest headers
r.headers['X-Custom-Token'] = self.token
return r
```
- **Why it is a good reference:** Shows how authentication handlers hook cleanly into `PreparedRequest` before network dispatch without tight coupling to the transport adapter.
19 changes: 19 additions & 0 deletions examples/requests/.codeintel/DIRECTORY_MAP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Directory Map: `psf/requests`

| Path | Purpose | Key Files & Entry Points |
| :--- | :--- | :--- |
| `src/requests/` | Core package source root. | `__init__.py` |
| `src/requests/api.py` | Top-level convenience functions (`get`, `post`, `request`). | `request()`, `get()`, `post()` |
| `src/requests/sessions.py` | State persistence, connection reuse, redirect loops, adapter mounting. | `Session`, `SessionRedirectMixin` |
| `src/requests/adapters.py` | Transport layer mapping `PreparedRequest` to `urllib3.PoolManager`. | `BaseAdapter`, `HTTPAdapter` |
| `src/requests/models.py` | Request/Response data models and data serialization. | `Request`, `PreparedRequest`, `Response` |
| `src/requests/auth.py` | Authentication handlers. | `AuthBase`, `HTTPBasicAuth`, `HTTPDigestAuth` |
| `src/requests/cookies.py` | CookieJar compatibility, extraction, and merging helpers. | `RequestsCookieJar`, `cookiejar_from_dict` |
| `src/requests/exceptions.py` | Exception hierarchy rooted in `RequestException`. | `RequestException`, `HTTPError`, `ConnectionError`, `Timeout` |
| `src/requests/structures.py` | Internal specialized data structures. | `CaseInsensitiveDict`, `LookupDict` |
| `src/requests/utils.py` | Common utility functions (URL quoting, header parsing, proxy discovery). | `get_environ_proxies`, `super_len`, `urldefragauth` |
| `src/requests/compat.py` | Python standard library compatibility shims. | `urlparse`, `str`, `bytes` imports |
| `src/requests/status_codes.py` | HTTP status code lookup dict (`codes.ok`, `codes.not_found`). | `_codes`, `codes` |
| `tests/` | Pytest test suite covering unit and integration scenarios. | `test_requests.py`, `conftest.py` |
| `tests/testserver/` | Threaded socket server used to test HTTP behavior without internet access. | `server.py` |
| `tests/certs/` | Test certificates (valid, expired, and mTLS certificates). | `certs/valid/`, `certs/expired/`, `certs/mtls/` |
34 changes: 34 additions & 0 deletions examples/requests/.codeintel/FEATURE_IMPLEMENTATION_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Feature Implementation Guide: `psf/requests`

Use this guide when adding features or fixing bugs in `psf/requests` to adhere to project invariants.

---

## 1. Locate Owning Layer
- Adding a high-level user API parameter (e.g. new keyword arg) ➔ `requests.api`, `requests.sessions.Session.request`, and `requests.models.Request`.
- Modifying how headers/body/params are prepared ➔ `requests.models.PreparedRequest.prepare_*`.
- Modifying transport, retries, SSL verification, or proxies ➔ `requests.adapters.HTTPAdapter`.
- Adding auth mechanism ➔ `requests.auth.AuthBase`.

---

## 2. Parameter Propagation Invariant
When adding any new parameter to `requests.get()` or `requests.post()`:
1. Add it to `requests/api.py` in the appropriate wrapper function and propagate to `session.request()`.
2. Update `Session.request()` in `src/requests/sessions.py` to accept the parameter.
3. Pass it to `Session.prepare_request()` or `Session.send()` depending on whether it affects the payload or transport.
4. If it affects the transport, ensure `HTTPAdapter.send()` in `src/requests/adapters.py` accepts and processes it.

---

## 3. Preserving Backwards Compatibility
- Never remove or re-order positional arguments on public methods.
- Default all new options to values that preserve existing behavior (e.g. `feature_enabled=False` or `None`).
- Return types must remain `Response` instances.

---

## 4. Testing Checklist
- Add unit tests in `tests/test_requests.py` verifying both default behavior and new option behavior.
- Ensure all tests run offline without remote internet access.
- Run `pytest` locally to confirm 0 regressions across the existing test suite.
Loading
Loading