Skip to content

docs(server): explain streaming request interruptions - #2391

Open
original4422 wants to merge 1 commit into
abetlen:mainfrom
original4422:docs/server-request-interruption
Open

original4422 wants to merge 1 commit into
abetlen:mainfrom
original4422:docs/server-request-interruption

Conversation

@original4422

Copy link
Copy Markdown

Summary

Explain the existing server request interruption behavior, as requested in #897:

 Server options
+ Streaming interruptions: default True, queued mode with False, and serialization limits

Add CLI and top-level JSON examples for letting the current stream finish before waiting requests run.
Clarify that requests still share a model proxy lock, and client disconnections or other cancellation can still end a stream.
This documents existing behavior; it does not change defaults or implement parallel decoding.

Evidence

  • Before: the server page had a generated settings reference but no explanation of stream interruption or queueing.
    After: the new section and its CLI/JSON examples render in the actual MkDocs page, with settings anchors preserved.
  • Original-package CLI parsing and Settings, ServerSettings and ConfigFileSettings checks confirm the documented True default and explicit false values.
  • python -m mkdocs build --clean and make lint pass. The baseline's 8 focused original tests also pass.
  • Existing MkDocs deprecation/link/Griffe warnings remain. Full model-download tests and inference/GPU/concurrency performance tests were not run for this documentation change.

AI-assisted documentation change.

Merge Danger

Door: two-way

Blast Radius: documentation

One server documentation page changes; runtime code and dependencies stay unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant