feat(llm): track, propagate, and log LLM token usage across evolution…

[rramosp]

Summary

This PR introduces end-to-end tracking, propagation, persistence, and logging of LLM token usage (prompt_tokens, completion_tokens, and total_tokens) across OpenEvolve runs. Token metrics are captured directly from OpenAI-compatible API responses, propagated from worker processes back to the main controller, stored in program metadata and prompt logs, and reported both per-iteration and as a cumulative summary at the end of evolution.

Key Changes

1. LLM & Ensemble Token Capture

• OpenAILLM (openevolve/llm/openai.py):
  • Added self.last_usage attribute to store the most recent call's token breakdown (prompt_tokens, completion_tokens, total_tokens, and model).
  • Extracted usage statistics from response.usage in _generate_chat(), with a fallback to prompt_tokens + completion_tokens if total_tokens is not explicitly provided.
  • Added INFO-level logging per API call:
    LLM Token Usage (<model>): prompt_tokens=..., completion_tokens=..., total_tokens=... (or logs when usage data is omitted by the provider).
• LLMEnsemble (openevolve/llm/ensemble.py):
  • Added self.last_usage and updated generate() and generate_with_context() to propagate last_usage from whichever underlying model was sampled for the request.

2. Worker-to-Controller Propagation & Logging

• SerializableResult & Worker (openevolve/process_parallel.py):
  • Added token_usage: Optional[Dict[str, Any]] = None to SerializableResult.
  • In _run_iteration_worker(), captured _worker_llm_ensemble.last_usage immediately after LLM generation and attached it to all SerializableResult return paths—including failed iterations (e.g., None response, diff parsing errors, missing changes_description, or code length limit violations) so token costs from discarded generations are still accounted for.
  • Included token_usage in the child Program metadata dictionary.
• ProcessParallelController (openevolve/process_parallel.py):
  • Aggregated total_prompt_tokens, total_completion_tokens, total_tokens, and total_llm_calls across all completed worker futures.
  • Appended token counts (| tokens: <total> (prompt: <prompt>, completion: <completion>)) to both successful iteration completion logs and iteration warning/error logs.
  • Added a cumulative summary log at the conclusion of the evolution run:
    📊 Total LLM Token Usage (<N> calls): total_tokens=... (prompt_tokens=..., completion_tokens=...)

3. Database & Prompt Log Persistence

• ProgramDatabase (openevolve/database.py):
  • Updated log_prompt() to accept token_usage: Optional[Dict[str, Any]] = None (and broadened the prompt dict type annotation from Dict[str, str] to Dict[str, Any]).
  • Recorded token_usage alongside prompt/response entries when log_prompts is enabled.

4. Unit Tests

• Unit Tests (tests/test_llm_ensemble.py):
  • Added TestTokenUsageLogging (IsolatedAsyncioTestCase) to verify that OpenAILLM accurately parses response.usage, populates last_usage, emits the expected INFO log, and that LLMEnsemble forwards last_usage upon generation.

Testing

• Ran unit test suite (pytest tests/test_llm_ensemble.py), confirming token extraction, log formatting, and ensemble propagation pass.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

raul ramos seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}