diff --git a/CHANGELOG.md b/CHANGELOG.md index 0517ee6..05b8035 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,23 @@ All notable changes to lcode are documented here. The format follows ## [Unreleased] +### Fixed + +- **The line after a load counts the part of the model in system RAM.** With Ollama 0.40, `qwen3.6-35b` + at 32K on a 12 GB GPU showed as `10.5 GB, 98% on the GPU and the rest in system RAM`, because Ollama + leaves out the ~12 GB of experts that stay mapped from the model's file. For the models in the catalog, + lcode now counts the whole model at its estimate, the one `lcode setup` and `lcode doctor` show: + `~23 GB, 42% on the GPU and the rest in system RAM`. With Ollama's numbers, the warning that another + program is using GPU memory couldn't fire. Now it does, and it says how much of the model fits on the GPU. +- **`lcode doctor` shows the loaded model.** It looked for `lcode-qwen3.6-35b` while Ollama lists it as + `lcode-qwen3.6-35b:latest`, so the `Loaded` line never appeared for the models `lcode setup` creates. + It now says how the model fits, in the same words as the line after a load. +- **`lcode bench` counts the part of the model in system RAM, too.** For `qwen3.6-35b` at 32K on a + 12 GB GPU, it said `10.4 GB, 98% on the GPU` after loading, and the same in its summary. It now says + `~23 GB, 42% on the GPU and the rest in system RAM`, as the line after a load does. `--json` results + keep Ollama's count in `memory_gb` and `gpu_percent`, so they still compare with older results, and + add lcode's estimate in `estimated_memory_gib` and `estimated_gpu_percent`. + ## [0.62.0] - 2026-11-06 ### Changed diff --git a/docs/bench.md b/docs/bench.md index 9260044..ef14788 100644 --- a/docs/bench.md +++ b/docs/bench.md @@ -25,7 +25,7 @@ its memory use, so those columns stay empty. lcode bench · NVIDIA GeForce RTX 4080 Laptop GPU (12 GB VRAM), 31 GB RAM · Ollama 0.40.13 lcode-qwen3.6-35b · 32K context · reasoning on · 8 tasks · 3 attempts each - Loaded in 18s · 10.4 GB, 98% on the GPU · reads prompts at 905 tok/s + Loaded in 18s · ~23 GB, 42% on the GPU and the rest in system RAM · reads prompts at 905 tok/s Attempt 1 of 3 ✓ fix-bug Fix a bug so the tests pass 22s the tests pass ✓ find-code Answer with file:line 8s next_wait at backoff.py:13 @@ -182,7 +182,7 @@ quick tasks once, as `lcode bench` did before 0.57. | **Generation** | Output tokens per second, across all the model's responses | | **Prompt reading** | Prompt tokens per second: how fast the model reads files and tool output. Measured per model with fresh 4,000-token prompts (the faster of two readings, since the first one after loading includes warm-up), because during the tasks Ollama serves most of each request from its prompt cache | | **Tool-call errors** | Tool calls that failed (wrong arguments, editing without reading first, unmatched edits) or that the model wrote as invalid JSON | -| **Memory** | The model's memory use and how much of it is on the GPU, from Ollama | +| **Memory** | The model's memory and how much of it is on the GPU, counted as in the [line after a load](usage.md#whats-happening-under-the-hood). Ollama leaves out what stays mapped from the model's file in system RAM, so for a model in `lcode models` that's partly off the GPU, it's lcode's estimate of the whole model (`~23 GB`). Otherwise it's what Ollama says | Results depend on the context window, so `lcode bench` uses **32K** by default on every machine, which makes results comparable. When comparing models, each one is unloaded before the next starts @@ -217,7 +217,7 @@ RTX 4080 Laptop GPU (12 GB) with 31 GB of RAM, Ollama 0.40.13, 32K context, reas | Generation | 64.0 tok/s | | Prompt reading | 851 tok/s | | Tool-call errors | 4 | -| Memory | 10.5 GB, 98% on GPU | +| Memory | ~23 GB, 42% on GPU | What it missed: - one `spec-fix` attempt fixed the reported case but not all the docstring's rules; @@ -226,6 +226,9 @@ What it missed: - every `runtime-bug` attempt settled for a band-aid: the window still grew, or still couldn't be made smaller. +The table shows memory as lcode counts it now. lcode 0.58.0 printed Ollama's count, +`10.5 GB, 98% on GPU`, which leaves out the ~12 GB of experts in system RAM. + Two `wide-feature` attempts were first counted as failed. Their `budget list` printed `No budgets set.` without budgets, and one listed the budgeted categories first. The task asks for neither behavior, so the check now accepts both (0.59.0), and the table counts them as passed. @@ -302,6 +305,8 @@ existing fields keep their meaning until `schema` changes. "prompt_tps": 904.6, "memory_gb": 10.4, "gpu_percent": 98, + "estimated_memory_gib": 22.7, + "estimated_gpu_percent": 42, "attempts": 3, "passed": 24, "total": 24, @@ -328,7 +333,8 @@ existing fields keep their meaning until `schema` changes. | `ollama`, `server` | The Ollama version, or `null` with [another model server](servers.md); `server` names the server either way | | `runs[].model` | The name you gave (catalog key or Ollama tag); `ollama_model` is the model that ran | | `runs[].context`, `num_batch`, `think` | The settings used. lcode lowers the batch size or context if the GPU runs out of memory, and reports what it ended up using | -| `runs[].memory_gb`, `gpu_percent` | From Ollama's `/api/ps`; `null` if unavailable | +| `runs[].memory_gb`, `gpu_percent` | The model's memory in GB and the share of it on the GPU, as Ollama's `/api/ps` counts them; `null` if unavailable. Ollama 0.40 leaves out what stays mapped from the model's file in system RAM: `qwen3.6-35b` at 32K on a 12 GB GPU shows as 10.4 GB, 98% of it on the GPU | +| `runs[].estimated_memory_gib`, `estimated_gpu_percent` | When Ollama's count leaves out part of a model in `lcode models`: lcode's estimate of the whole model in GiB, and the share of it on the GPU, as the summary shows them (`~23 GB, 42% on GPU`). `null` when Ollama counts the whole model (all of it on the GPU, or a model outside the catalog) and in older results | | `runs[].generation_tps` | Output tokens per second across the tasks; `null` if Ollama reported no timings | | `runs[].prompt_tps` | Prompt tokens per second for a fresh 4,000-token prompt (the faster of two readings); `null` if it couldn't be measured | | `runs[].usage` | Totals over the tasks as Ollama reports them. `prompt_tokens` includes tokens served from the prompt cache | diff --git a/docs/models.md b/docs/models.md index 69c519e..ff150d0 100644 --- a/docs/models.md +++ b/docs/models.md @@ -258,7 +258,8 @@ RAM needed ≈ memory − GPU memory + ~10 GB for Ollama's server and prompt cac - **The part of the model that doesn't fit on the GPU.** A Mixture-of-Experts model's experts stay mapped from the model's file, so they have to stay in Linux's page cache. `ollama ps` leaves them out: - it shows `qwen3.6-35b` at 32K as 10.5 GB, with ~12 GB more in RAM. + it shows `qwen3.6-35b` at 32K as 10.5 GB, with ~12 GB more in RAM. lcode counts them: after a load, + in `lcode doctor` and in `lcode bench`, it says `~23 GB, 42% on the GPU and the rest in system RAM`. - **Ollama's prompt cache**, which grows to 8 GB in a long session, next to 1–2 GB of the server's own. On a 12 GB GPU, `qwen3.6-35b` needs ~21 GB of RAM at 32K, ~23 GB at 128K and ~26 GB at 256K. On Linux, diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 0cd6a30..f904b4a 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -84,8 +84,10 @@ need `lcode mcp login `. More in [MCP servers: Troubleshooting](mcp.md#tro starts, and `keep_alive` keeps it loaded between requests. - Reading many large files takes a while the first time (~280 tokens/s on a 12 GB GPU, ~500 with `lcode config set num_batch 1024` if it fits); follow-up turns reuse the cache. -- `ollama ps` shows how much of the model is on the GPU. Dense models are much slower when split; - prefer the MoE models in `lcode models` on smaller GPUs. +- `lcode doctor` shows how much of the loaded model is on the GPU, for example + `Loaded ~23 GB, 42% on the GPU and the rest in system RAM · 32K context`. `ollama ps` leaves out the + part of a Mixture-of-Experts model that's in system RAM, so it can say 98%. Dense models are much + slower when split; prefer the MoE models in `lcode models` on smaller GPUs. - `--no-think` skips reasoning for simple requests. ### It starts fast, then gets about three times slower diff --git a/docs/usage.md b/docs/usage.md index ebefe29..eeb4866 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -409,7 +409,7 @@ stays loaded: | When | What it says | |---|---| -| The model was loaded | `◆ Loaded qwen3.6-35b in 15s: 22.3 GB, 45% on the GPU and the rest in system RAM.`, and a warning when less of it is on the GPU than usual, because another program or model is using GPU memory | +| The model was loaded | `◆ Loaded qwen3.6-35b in 15s: ~23 GB, 42% on the GPU and the rest in system RAM.`, and a warning when less of it is on the GPU than usual, because another program or model is using GPU memory | | A dense model doesn't fit on the GPU (at startup and on `/model`) | `⚠ qwen3.8-27b is a dense model: all 27B parameters work on every token. It's capable, but slow here. Only about 61% of it fits in the 12 GB of GPU memory…`, with what to do instead | | The model that looks at images needs the memory | `Loading the vision model … to look at bug.png; qwen3.6-35b makes room and reloads afterwards`, then `◆ qwen3.6-35b was unloaded to make room…`, and the next step says `Reloading qwen3.6-35b into memory (it made room for the vision model …)` | | A tool needs the GPU to itself (an MCP server with `free_gpu`) | `◆ Unloaded qwen3.6-35b so the server can use the GPU; it reloads afterwards.` | @@ -421,6 +421,12 @@ While the model loads in the background at startup, the status bar says part of them in system RAM. Dense ones slow down a lot, to a few tokens a second, which is why they get the warning. +The `~` marks lcode's estimate of the model's memory, the same one `lcode setup` and `lcode doctor` +show. Ollama leaves out what stays mapped from the model's file in system RAM: `ollama ps` shows +qwen3.6-35b at 32K on a 12 GB GPU as 10.5 GB, 98% of it on the GPU, while ~12 GB of its experts are +in RAM. So when part of a model in `lcode models` is off the GPU, lcode counts all of it. For other +models it repeats what Ollama says. `lcode doctor` shows the same line for the model in memory. + ## Steer while it works A request can take minutes with a local model. You don't have to stop it to change course: type diff --git a/src/lcode/bench.py b/src/lcode/bench.py index 03dc232..e972ad4 100644 --- a/src/lcode/bench.py +++ b/src/lcode/bench.py @@ -27,12 +27,12 @@ from rich.table import Table from rich.text import Text -from lcode import __version__, escalate +from lcode import __version__, escalate, residency from lcode.agent import Agent, Settings from lcode.bench_checks import Check, Task from lcode.bench_quick import TASKS from lcode.config import format_tokens -from lcode.hardware import Hardware +from lcode.hardware import GIB, Hardware from lcode.ollama import Ollama, OllamaError SCHEMA_VERSION = 1 @@ -188,8 +188,10 @@ class ModelRun: think: bool = True load_seconds: float = 0.0 prompt_tps: float | None = None # measured with a fresh prompt; see prompt_speed() - memory_gb: float | None = None - gpu_percent: int | None = None + memory_gb: float | None = None # as Ollama's /api/ps counts it, which can leave part of the model out + gpu_percent: int | None = None # of memory_gb, on the GPU + estimated_memory_gib: float | None = None # all of the model by the catalog's estimate, when Ollama leaves part out + estimated_gpu_percent: int | None = None # of estimated_memory_gib, on the GPU usage: dict = field(default_factory=dict) tasks: list[TaskResult] = field(default_factory=list) error: str | None = None # the model couldn't run at all @@ -219,6 +221,14 @@ def generation_tps(self) -> float | None: def tool_errors(self) -> int: return sum(t.tool_errors for t in self.tasks) + def note_memory(self, where: residency.Placement) -> None: + """Ollama's count keeps the names it always had, so results compare with older ones. The estimate that the line + after the load shows goes next to it.""" + self.memory_gb, self.gpu_percent = round(where.size / 1e9, 1), round(100 * where.vram / where.size) + if where.estimate > where.size: + self.estimated_memory_gib = round(where.estimate / GIB, 1) + self.estimated_gpu_percent = round(100 * where.gpu_share) + def to_json(self) -> dict: data = asdict(self) rates = self.rates() @@ -410,13 +420,11 @@ def _read_prompt(ollama: Ollama, settings: Settings, options: dict) -> float | N return round(count / (ns / 1e9), 1) if ns and count >= 1000 else None -def memory_use(ollama: Ollama, model: str) -> tuple[float | None, int | None]: - for entry in ollama.running(): - if entry.get("name") in (model, f"{model}:latest") or entry.get("model") in (model, f"{model}:latest"): - size, vram = entry.get("size") or 0, entry.get("size_vram") or 0 - if size: - return round(size / 1e9, 1), round(100 * vram / size) - return None, None +def memory_use(ollama: Ollama, model: str, context: int) -> residency.Placement | None: + """How the loaded model sits in memory, counted as the line after a load counts it: Ollama 0.40 leaves out the + part of a Mixture-of-Experts model that stays mapped from its file in RAM (see residency.with_estimate).""" + where = residency.placement(ollama, model) # Ollama lists lcode-qwen3.6-35b as lcode-qwen3.6-35b:latest + return residency.with_estimate(where, model, context) if where and where.size else None def run_model( @@ -455,14 +463,15 @@ def run_model( run.interrupted = True return run run.load_seconds = round(time.monotonic() - started, 1) - run.memory_gb, run.gpu_percent = memory_use(ollama, settings.model) + if placed := memory_use(ollama, settings.model, settings.context): + run.note_memory(placed) try: with console.status(Elapsed(" Measuring prompt speed")): run.prompt_tps = prompt_speed(ollama, settings, options) except KeyboardInterrupt: run.interrupted = True return run - where = f" · {run.memory_gb} GB, {run.gpu_percent}% on the GPU" if run.memory_gb else "" + where = f" · {placed.describe()}" if placed else "" reads = f" · reads prompts at {run.prompt_tps:,.0f} tok/s" if run.prompt_tps else "" if getattr(ollama, "kind", "ollama") == "ollama": console.print(f" [dim]Loaded in {run.load_seconds:.0f}s{where}{reads}[/]") @@ -503,8 +512,8 @@ def run_model( run.pruned, run.compacted = pruned + shared.pruned, compacted + shared.compacted if run.interrupted: break - if run.memory_gb is None: - run.memory_gb, run.gpu_percent = memory_use(ollama, settings.model) + if run.memory_gb is None and (placed := memory_use(ollama, settings.model, settings.context)): + run.note_memory(placed) # lcode lowers the batch size or context if the GPU runs out of memory; report what was used. run.context, run.num_batch, run.think = settings.context, settings.num_batch, settings.think return run @@ -521,6 +530,13 @@ def percent(share: float | None) -> str: return "–" if share is None else f"{share * 100:.0f}%" +def memory(run: ModelRun) -> str: + """'~23 GB, 42% on GPU' by lcode's estimate, as the line after the load counts it, or else what Ollama says.""" + if run.estimated_memory_gib: + return f"~{run.estimated_memory_gib:.0f} GB, {run.estimated_gpu_percent}% on GPU" + return f"{run.memory_gb} GB, {run.gpu_percent}% on GPU" if run.memory_gb else "?" + + def summary_rows(runs: list[ModelRun], tasks: list[Task]) -> list[tuple[str, list[str]]]: rows = [] for task in {t.id: t for t in tasks}.values(): # a task that ran several times: one row, a mark each @@ -554,10 +570,7 @@ def cell(run: ModelRun, value: str) -> str: ("Generation", [cell(r, f"{r.generation_tps:.1f} tok/s" if r.generation_tps else "?") for r in runs]), ("Prompt reading", [cell(r, f"{r.prompt_tps:,.0f} tok/s" if r.prompt_tps else "?") for r in runs]), ("Tool-call errors", [cell(r, str(r.tool_errors)) for r in runs]), - ( - "Memory", - [cell(r, f"{r.memory_gb} GB, {r.gpu_percent}% on GPU" if r.memory_gb else "?") for r in runs], - ), + ("Memory", [cell(r, memory(r)) for r in runs]), ("Context", [cell(r, format_tokens(r.context)) for r in runs]), ] if any(r.session for r in runs): diff --git a/src/lcode/doctor.py b/src/lcode/doctor.py index cf75918..9bfb1ae 100644 --- a/src/lcode/doctor.py +++ b/src/lcode/doctor.py @@ -110,15 +110,9 @@ def line(label: str, value: str, good: bool | None = True) -> None: f"here (delete {limits.path()} to try again)", None, ) - loaded = [m for m in ollama.running() if m.get("name") == model or m.get("model") == model] - if loaded: - m = loaded[0] - line( - "Loaded", - f"{m.get('size_vram', 0) / 1e9:.1f} GB on GPU of {m.get('size', 0) / 1e9:.1f} GB, " - f"context {m.get('context_length', '?')}", - True, - ) + if where := residency.placement(ollama, model): # Ollama lists lcode-qwen3.6-35b as lcode-qwen3.6-35b:latest + where = residency.with_estimate(where, model, ctx) + line("Loaded", where.describe() + (f" · {format_tokens(where.context)} context" if where.context else "")) except cli.NotInstalled as e: line("Model", str(e), False) ok = False diff --git a/src/lcode/residency.py b/src/lcode/residency.py index b010777..4f3f965 100644 --- a/src/lcode/residency.py +++ b/src/lcode/residency.py @@ -12,17 +12,18 @@ import threading import time -from dataclasses import dataclass +from dataclasses import dataclass, replace from statistics import median from urllib.parse import urlsplit from lcode import catalog from lcode.cloud import is_cloud from lcode.config import format_tokens -from lcode.hardware import Hardware +from lcode.hardware import GIB, Hardware from lcode.ollama import OllamaError GB = 1e9 +ALL = 0.995 # this share of a model on the GPU counts as all of it, and 1 - ALL as none of it RELEASE_WAIT = 15.0 # seconds to wait for Ollama to give an unloaded model's GPU memory back SPLIT_WARNING = 0.95 # a dense model with less than this on the GPU runs much slower SHORTFALL = 0.25 # this much less on the GPU than expected: something else is using GPU memory @@ -36,24 +37,30 @@ @dataclass class Placement: - """A loaded model, from Ollama's /api/ps.""" + """A loaded model, from Ollama's /api/ps (see `with_estimate` for what that leaves out).""" name: str - size: float # bytes in memory + size: float # bytes in memory, as Ollama counts them vram: float # bytes of it on the GPU context: int = 0 # the context window it was loaded with + estimate: float = 0.0 # bytes the model takes by the catalog's estimate, when Ollama's count leaves part out + + @property + def total(self) -> float: + return max(self.size, self.estimate) @property def gpu_share(self) -> float: - return self.vram / self.size if self.size else 0.0 + return self.vram / self.total if self.total else 0.0 def describe(self) -> str: - """'22.3 GB, 44% on the GPU and the rest in system RAM'""" - size = f"{self.size / GB:.1f} GB" + """'6.6 GB, all on the GPU', or with the catalog's estimate: '~23 GB, 42% on the GPU and the rest in system + RAM'. The estimate is shown as `lcode setup` and `lcode doctor` show it.""" + size = f"~{self.estimate / GIB:.0f} GB" if self.estimate > self.size else f"{self.size / GB:.1f} GB" share = self.gpu_share - if share >= 0.995: + if share >= ALL: return f"{size}, all on the GPU" - if share <= 0.005: + if share <= 1 - ALL: return f"{size}, all in system RAM (on the CPU)" return f"{size}, {share:.0%} on the GPU and the rest in system RAM" @@ -117,6 +124,20 @@ def placement(ollama, model: str) -> Placement | None: return found[0] if found else None +def with_estimate(where: Placement, model: str, context: int) -> Placement: + """`where`, counting the whole model when part of it is off the GPU: by the catalog's estimate (models outside + the catalog stay as Ollama counts them). + + /api/ps counts the memory Ollama's runner allocates. The weights it keeps in system RAM aren't allocated: they + stay mapped from the model's file. So with Ollama 0.40, qwen3.6-35b at 32K on a 12 GB GPU shows as 10.5 GB, 98% + of it on the GPU, while ~12 GB of its experts are in RAM. A model all on the GPU is counted in full. + """ + spec = catalog.find(model) + if spec is None or where.vram >= ALL * where.size: + return where + return replace(where, estimate=spec.memory_gib(where.context or context) * GIB) + + def is_moe(ollama, model: str) -> bool | None: """Whether the model is a Mixture-of-Experts (None: unknown).""" spec = catalog.find(model) @@ -169,14 +190,17 @@ def load_report( ) -> str: """The line after a load: how long it took, how it fits, and a warning if it fits worse than it should.""" took = f" in {seconds:.0f}s" if seconds >= 1 else "" - line = f"{'Reloaded' if reload else 'Loaded'} {model}{took}" + (f": {where.describe()}." if where else ".") + line = f"{'Reloaded' if reload else 'Loaded'} {model}{took}" if where is None: - return line + return f"{line}." + where = with_estimate(where, model, context) + line += f": {where.describe()}." expected = expected_share(model, context, hardware) if expected is not None and where.gpu_share < expected - SHORTFALL: + fits = "all of it fits" if expected >= ALL else f"up to {expected:.0%} fits" line += ( - f" Less of it is on the GPU than usual ({expected:.0%}): another program or model is using GPU " - "memory, so it runs slower until that's closed." + f" Less of it is on the GPU than usual ({fits}): another program or model is using GPU memory, so it " + "runs slower until that's closed." ) elif dense and where.gpu_share < SPLIT_WARNING: line += " It's a dense model split between GPU and CPU, so expect it to be slow." diff --git a/tests/test_bench.py b/tests/test_bench.py index 3131566..fcc03e3 100644 --- a/tests/test_bench.py +++ b/tests/test_bench.py @@ -1,3 +1,4 @@ +import dataclasses import io import json import subprocess @@ -263,6 +264,64 @@ def test_report_and_markdown(tmp_path): assert "| fix-bug: Fix a bug so the tests pass | ✓ " in md +# What Ollama 0.40.13 lists for lcode-qwen3.6-35b at 32K on a 12 GB laptop: the ~12 GB of experts that stay mapped +# from the model's file in RAM aren't in its size, so 98% of it seems to be on the GPU. +OLLAMA_0_40 = { + "name": "lcode-qwen3.6-35b:latest", + "model": "lcode-qwen3.6-35b:latest", + "size": 10_490_000_000, + "size_vram": 10_290_000_000, + "context_length": 32768, +} +MEMORY = ("memory_gb", "gpu_percent", "estimated_memory_gib", "estimated_gpu_percent") + + +def bench_loaded(model: str, entry: dict) -> tuple[bench.ModelRun, str]: + """`model` benched without tasks while Ollama lists `entry` as loaded: the run, and what it printed.""" + ollama = FakeOllama([probe(), probe()]) + ollama.loaded = [entry] + console = quiet() + run = bench.run_model(ollama, model, dataclasses.replace(settings(), model=model), [], console) + return run, console.file.getvalue() + + +def saved_memory(run: bench.ModelRun) -> list: + return [bench.report([run], LAPTOP, "0.40.13")["runs"][0][key] for key in MEMORY] + + +def test_memory_counts_the_part_of_the_model_ollama_leaves_out(): + run, printed = bench_loaded("lcode-qwen3.6-35b", OLLAMA_0_40) # Ollama lists it with :latest + assert "Loaded in 0s · ~23 GB, 42% on the GPU and the rest in system RAM · reads prompts at 2,000 tok/s" in printed + assert (run.memory_gb, run.gpu_percent) == (10.5, 98) # what Ollama says, under the names older results use + assert (run.estimated_memory_gib, run.estimated_gpu_percent) == (22.7, 42) # the ~23 GB lcode setup shows + assert saved_memory(run) == [10.5, 98, 22.7, 42] + assert "| Memory | ~23 GB, 42% on GPU |" in bench.markdown([run], [], LAPTOP, "0.40.13") + + +def test_memory_is_what_ollama_says_when_it_counts_the_whole_model(): + run, printed = bench_loaded("qwen3.5-9b", {"name": "qwen3.5-9b", "size": 6.6e9, "size_vram": 6.6e9}) + assert "Loaded in 0s · 6.6 GB, all on the GPU · reads" in printed + assert saved_memory(run) == [6.6, 100, None, None] + assert "| Memory | 6.6 GB, 100% on GPU |" in bench.markdown([run], [], LAPTOP, "0.40.13") + run, printed = bench_loaded("custom:7b", {**OLLAMA_0_40, "name": "custom:7b", "model": "custom:7b"}) # no estimate + assert "Loaded in 0s · 10.5 GB, 98% on the GPU and the rest in system RAM · reads" in printed + assert saved_memory(run) == [10.5, 98, None, None] and bench.memory(run) == "10.5 GB, 98% on GPU" + + +def test_memory_is_measured_after_the_tasks_when_the_model_loaded_late(monkeypatch): + ollama = FakeOllama() + ollama.loaded = [] # the load failed: the model isn't in memory when it's measured after loading + + def measure(*args): # a later request loads it + ollama.loaded = [OLLAMA_0_40] + + monkeypatch.setattr(bench, "prompt_speed", measure) + console = quiet() + run = bench.run_model(ollama, "qwen3.6-35b", settings(), [], console) + assert "GB" not in console.file.getvalue() + assert saved_memory(run) == [10.5, 98, 22.7, 42] + + def test_select_tasks(): quick = [t.id for t in bench.TASKS] longer = ["cli-tool", "flaky-test", "repo-bug", "wide-feature", "fast-dedupe", "runtime-bug"] diff --git a/tests/test_residency.py b/tests/test_residency.py index 59482ec..1c23865 100644 --- a/tests/test_residency.py +++ b/tests/test_residency.py @@ -21,6 +21,11 @@ def entry(name, size_gb, gpu_gb, context=0): return {"name": name, "size": size_gb * 1e9, "size_vram": gpu_gb * 1e9, "context_length": context} +# What Ollama 0.40.13 lists for lcode-qwen3.6-35b at 32K on the laptop: the ~12 GB of experts that stay mapped from +# the model's file in RAM aren't in its size, so 98% of it seems to be on the GPU. +OLLAMA_0_40 = {**entry("lcode-qwen3.6-35b:latest", 10.49, 10.29, 32768), "model": "lcode-qwen3.6-35b:latest"} + + def text(agent): return " ".join(output(agent).split()) @@ -31,6 +36,24 @@ def test_where_a_model_sits(): assert Placement("m", 6e9, 0).describe() == "6.0 GB, all in system RAM (on the CPU)" +def estimated(model, size_gb, gpu_gb, context=32768, fallback=32768): + return residency.with_estimate(Placement(model, size_gb * 1e9, gpu_gb * 1e9, context), model, fallback).describe() + + +def test_the_part_of_a_catalog_model_ollama_leaves_out_is_counted(): + assert estimated("lcode-qwen3.6-35b", 10.49, 10.29) == "~23 GB, 42% on the GPU and the rest in system RAM" + assert f"~{catalog.find('qwen3.6-35b').memory_gib(32768):.0f} GB" == "~23 GB" # as lcode setup and doctor say + assert estimated("qwen3.6-35b", 10.49, 10.29, context=262144) == "~28 GB, 35% on the GPU and the rest in system RAM" + assert estimated("qwen3.6-35b", 10.49, 10.29, context=0) == "~23 GB, 42% on the GPU and the rest in system RAM" + assert estimated("qwen3.5-4b", 1.2, 0) == "~5 GB, all in system RAM (on the CPU)" # no GPU: all of it is in RAM + + +def test_what_ollama_counts_stands_when_nothing_is_left_out_or_nothing_is_known(): + assert estimated("qwen3.5-9b", 8.5, 8.5, context=65536) == "8.5 GB, all on the GPU" # below the estimate + assert estimated("qwen3.6-35b", 26, 10) == "26.0 GB, 38% on the GPU and the rest in system RAM" # above it + assert estimated("custom:7b", 10.49, 10.29) == "10.5 GB, 98% on the GPU and the rest in system RAM" # no catalog + + def test_dense_models_that_dont_fit_get_a_warning(make_agent): ollama = make_agent().ollama warning = residency.dense_warning(ollama, "qwen3.8-27b", 32768, LAPTOP) @@ -54,29 +77,42 @@ def test_models_outside_the_catalog_are_asked_about(make_agent): def test_a_load_is_reported_once_with_how_it_fits(make_agent): - agent = make_agent([reply("hi"), reply("again")], model="qwen3.6-35b", context=32768) + agent = make_agent([reply("hi"), reply("again")], context=32768) agent.hardware = LAPTOP agent.ollama.loaded = [] def chat_stream(payload, on_open=None): agent.ollama.payloads.append(payload) - agent.ollama.loaded = [entry("qwen3.6-35b:latest", 22.3, 9.8, 32768)] + agent.ollama.loaded = [OLLAMA_0_40] yield from agent.ollama.scripts.pop(0) agent.ollama.chat_stream = chat_stream agent.run_turn("hello") - assert "◆ Loaded qwen3.6-35b: 22.3 GB, 44% on the GPU and the rest in system RAM." in text(agent) + assert "◆ Loaded lcode-qwen3.6-35b: ~23 GB, 42% on the GPU and the rest in system RAM." in text(agent) + assert "⚠" not in output(agent) # as much is on the GPU as usual agent.console.file.truncate(0) agent.run_turn("and again") assert "◆" not in output(agent) and "⚠" not in output(agent) # in memory: nothing to say -def test_less_on_the_gpu_than_usual_is_explained(make_agent): - report = residency.load_report("qwen3.6-35b", 14, Placement("m", 22.3e9, 2e9), 32768, LAPTOP, dense=False) - assert report.startswith("Loaded qwen3.6-35b in 14s: 22.3 GB, 9% on the GPU") - assert "another program or model is using GPU memory" in report - dense = residency.load_report("qwen3.8-27b", 40, Placement("m", 18e9, 11e9), 32768, LAPTOP, dense=True) - assert dense.endswith("It's a dense model split between GPU and CPU, so expect it to be slow.") +def report(model, size_gb, gpu_gb, context=32768, dense=False): + placement = Placement("m", size_gb * 1e9, gpu_gb * 1e9, context) + return residency.load_report(model, 14, placement, context, LAPTOP, dense=dense) + + +def test_less_on_the_gpu_than_usual_is_explained(): + # Another program holds GPU memory: Ollama counts 4.2 GB, 95% of it on the GPU, and leaves out the rest in RAM. + assert report("qwen3.6-35b", 4.2, 4.0) == ( + "Loaded qwen3.6-35b in 14s: ~23 GB, 16% on the GPU and the rest in system RAM. Less of it is on the GPU than " + "usual (up to 53% fits): another program or model is using GPU memory, so it runs slower until that's closed." + ) + assert report("qwen3.6-35b", 10.49, 10.29) == ( + "Loaded qwen3.6-35b in 14s: ~23 GB, 42% on the GPU and the rest in system RAM." # the usual 9-10 GB + ) + assert "(all of it fits)" in report("qwen3.5-9b", 7.7, 7.5, context=131072, dense=True) + split = "It's a dense model split between GPU and CPU, so expect it to be slow." + assert report("qwen3.8-27b", 18, 11, dense=True).endswith(split) + assert report("qwen3.8-27b", 11.2, 11, dense=True).endswith(split) # with the part in RAM left out, too def test_the_status_line_says_what_comes_before_the_first_token(make_agent): @@ -390,6 +426,18 @@ def test_doctor_says_nothing_about_ram_it_cant_tell(cli_here, machine): assert "RAM " not in cli_here() +def test_doctor_shows_how_the_loaded_model_fits(cli_here, machine, monkeypatch): + machine(available=25.0, ollama_private=0.0) + ollama = LocalOllama() + ollama.loaded = [OLLAMA_0_40] # as lcode-qwen3.6-35b:latest, while the model resolves to lcode-qwen3.6-35b + monkeypatch.setattr(cli, "Ollama", lambda host: ollama) + assert run_doctor() == 0 + assert "✓ Loaded ~23 GB, 42% on the GPU and the rest in system RAM · 32K context\n" in cli_here() + ollama.loaded = [entry("qwen3.5-9b", 6.6, 6.6, 131072)] # another model + assert run_doctor() == 0 + assert cli_here().count("Loaded") == 1 + + def test_setup_warns_about_ram_on_its_recommendation(cli_here, machine, monkeypatch): machine(available=22.0, ollama_private=0.0) monkeypatch.setattr(cli.Confirm, "ask", lambda *args, **kwargs: False)