feat: add configurations

[kerthcet]

What this PR does / why we need it

Which issue(s) this PR fixes

Fixes #

Special notes for your reviewer

Does this PR introduce a user-facing change?

[Copilot]

Pull request overview

This PR introduces a configurable inference engine configuration surface and wires it through both the CLI and API so defaults (especially the max-tokens budget) are no longer hardcoded.

Changes:

• Add EngineConfig and plumb it into backend::engine(...) so KV-cache sizing, block sizing, batching, and default token budget are configurable.
• Add shared CLI engine-tuning flags for both puma run and puma serve, with docs describing the available options.
• Update OpenAI-compatible API endpoints to use the engine’s configured default max_tokens when requests omit it.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/cli/serve.rs	Passes an EngineConfig into engine construction for serve.
src/cli/commands.rs	Adds shared EngineArgs flags (flattened into run/serve) and tests for flag → config mapping.
src/backend/mod.rs	Re-exports EngineConfig from the backend module surface.
src/backend/llm_engine.rs	Introduces EngineConfig, threads it into engine(), and stores default_max_tokens on EngineHandle.
src/api/tests.rs	Updates API test harness to pass EngineConfig::default() into engine().
src/api/completions.rs	Uses the engine’s configured default token budget when max_tokens is omitted.
src/api/chat.rs	Uses the engine’s configured default token budget when max_tokens is omitted (streaming + non-streaming).
README.md	Documents new engine-tuning usage and updates example commands/models.
docs/configuration.md	Adds documentation for engine-tuning flags and defaults.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/cli/commands.rs:110

• tokens_per_block can currently be set to 0 via CLI, but SequenceManager::new asserts tokens_per_block > 0, so an invalid flag value will cause a runtime panic. Add a clap value parser/range to reject 0 at argument-parse time.

    /// Size of a single KV block, in bytes (must be > 0)
    #[arg(
        long,

src/cli/commands.rs:114

• max_batch_size can currently be set to 0 via CLI; in Scheduler::schedule_prefill this makes the prefill batch permanently "full" (len >= 0), so nothing ever schedules. Reject 0 at parse time with a clap range/value_parser.

        value_parser = clap::value_parser!(usize).range(1..)
    )]
    block_size_bytes: usize,

src/backend/llm_engine.rs:408

• engine() accepts EngineConfig from both library callers and the CLI, but there are no invariants enforced here. Invalid values (e.g., tokens_per_block == 0 triggers an assert in SequenceManager::new; max_batch_size == 0 stalls scheduling) will currently fail at runtime in less obvious places. Consider validating EngineConfig at the start of engine() to fail fast with a clear message.

    config: EngineConfig,
) -> (EngineHandle, EngineRunner<B>) {
    // KV-cache memory pool + block manager.
    let allocator = Box::new(CpuAllocator::new(config.memory_pool_bytes));
    let block_manager = BlockManager::new(allocator, config.block_size_bytes);

[kerthcet]

/lgtm
/kind feature

[InftyAI-Agent]

Approved: PR has both lgtm and approved labels

[InftyAI-Agent]

Approved: PR has both lgtm and approved labels

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/backend/llm_engine.rs:408

• EngineConfig documents constraints (e.g., tokens_per_block must be > 0), but engine() doesn’t validate them. Passing an invalid config can cause panics deeper in the stack (e.g., SequenceManager::new asserts tokens_per_block > 0). Adding early validation here makes the public API safer and the failure mode clearer.

    // KV-cache memory pool + block manager.
    let allocator = Box::new(CpuAllocator::new(config.memory_pool_bytes));
    let block_manager = BlockManager::new(allocator, config.block_size_bytes);

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

src/api/completions.rs:79

• engine.default_max_tokens() will never be used here if the request type still deserializes max_tokens with a default of Some(100). Currently ChatCompletionRequest/CompletionRequest in src/api/types/request.rs uses #[serde(default = "default_max_tokens")] which sets max_tokens to Some(100) when the field is omitted, so this change won’t make the engine-configured default effective.

    match engine
        .generate(
            &req.model,
            &prompt,
            req.max_tokens
                .unwrap_or_else(|| engine.default_max_tokens()),
            req.temperature.unwrap_or(0.7),

src/api/chat.rs:103

• This fallback to engine.default_max_tokens() is likely unreachable in practice because ChatCompletionRequest.max_tokens is deserialized with a default (#[serde(default = "default_max_tokens")] returning Some(100)). That means requests which omit max_tokens will still come through as Some(100), so the engine-configured default won’t apply.

    let response = engine
        .generate(
            &req.model,
            &prompt,
            req.max_tokens
                .unwrap_or_else(|| engine.default_max_tokens()),
            req.temperature.unwrap_or(0.7),
        )

src/api/chat.rs:176

• Same issue as the non-streaming path: req.max_tokens.unwrap_or_else(|| engine.default_max_tokens()) won’t use the engine default if the request struct is defaulting max_tokens to Some(100) on deserialization when the field is omitted.

        match engine
            .generate_stream(
                &model,
                &prompt,
                req.max_tokens
                    .unwrap_or_else(|| engine.default_max_tokens()),
                req.temperature.unwrap_or(0.7),
            )

src/backend/llm_engine.rs:408

• engine() now accepts a user-provided EngineConfig, but it doesn’t validate invariants (e.g., tokens_per_block > 0). Passing 0 will currently panic inside SequenceManager::new (and block_size_bytes == 0 can result in zero-sized allocations). Adding explicit checks here provides a clearer failure mode and prevents accidental invalid configs from crashing the engine in less obvious places.

    // KV-cache memory pool + block manager.
    let allocator = Box::new(CpuAllocator::new(config.memory_pool_bytes));
    let block_manager = BlockManager::new(allocator, config.block_size_bytes);