diff --git a/CHANGELOG.md b/CHANGELOG.md index 0517ee6..4965eb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,18 @@ All notable changes to lcode are documented here. The format follows ## [Unreleased] +### Fixed + +- **The line after a load counts the part of the model in system RAM.** With Ollama 0.40, `qwen3.6-35b` + at 32K on a 12 GB GPU showed as `10.5 GB, 98% on the GPU and the rest in system RAM`, because Ollama + leaves out the ~12 GB of experts that stay mapped from the model's file. For the models in the catalog, + lcode now counts the whole model at its estimate, the one `lcode setup` and `lcode doctor` show: + `~23 GB, 42% on the GPU and the rest in system RAM`. With Ollama's numbers, the warning that another + program is using GPU memory couldn't fire. Now it does, and it says how much of the model fits on the GPU. +- **`lcode doctor` shows the loaded model.** It looked for `lcode-qwen3.6-35b` while Ollama lists it as + `lcode-qwen3.6-35b:latest`, so the `Loaded` line never appeared for the models `lcode setup` creates. + It now says how the model fits, in the same words as the line after a load. + ## [0.62.0] - 2026-11-06 ### Changed diff --git a/docs/models.md b/docs/models.md index 69c519e..9fbf93a 100644 --- a/docs/models.md +++ b/docs/models.md @@ -258,7 +258,8 @@ RAM needed ≈ memory − GPU memory + ~10 GB for Ollama's server and prompt cac - **The part of the model that doesn't fit on the GPU.** A Mixture-of-Experts model's experts stay mapped from the model's file, so they have to stay in Linux's page cache. `ollama ps` leaves them out: - it shows `qwen3.6-35b` at 32K as 10.5 GB, with ~12 GB more in RAM. + it shows `qwen3.6-35b` at 32K as 10.5 GB, with ~12 GB more in RAM. lcode counts them: after a load, + and in `lcode doctor`, it says `~23 GB, 42% on the GPU and the rest in system RAM`. - **Ollama's prompt cache**, which grows to 8 GB in a long session, next to 1–2 GB of the server's own. On a 12 GB GPU, `qwen3.6-35b` needs ~21 GB of RAM at 32K, ~23 GB at 128K and ~26 GB at 256K. On Linux, diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 0cd6a30..f904b4a 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -84,8 +84,10 @@ need `lcode mcp login `. More in [MCP servers: Troubleshooting](mcp.md#tro starts, and `keep_alive` keeps it loaded between requests. - Reading many large files takes a while the first time (~280 tokens/s on a 12 GB GPU, ~500 with `lcode config set num_batch 1024` if it fits); follow-up turns reuse the cache. -- `ollama ps` shows how much of the model is on the GPU. Dense models are much slower when split; - prefer the MoE models in `lcode models` on smaller GPUs. +- `lcode doctor` shows how much of the loaded model is on the GPU, for example + `Loaded ~23 GB, 42% on the GPU and the rest in system RAM · 32K context`. `ollama ps` leaves out the + part of a Mixture-of-Experts model that's in system RAM, so it can say 98%. Dense models are much + slower when split; prefer the MoE models in `lcode models` on smaller GPUs. - `--no-think` skips reasoning for simple requests. ### It starts fast, then gets about three times slower diff --git a/docs/usage.md b/docs/usage.md index ebefe29..eeb4866 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -409,7 +409,7 @@ stays loaded: | When | What it says | |---|---| -| The model was loaded | `◆ Loaded qwen3.6-35b in 15s: 22.3 GB, 45% on the GPU and the rest in system RAM.`, and a warning when less of it is on the GPU than usual, because another program or model is using GPU memory | +| The model was loaded | `◆ Loaded qwen3.6-35b in 15s: ~23 GB, 42% on the GPU and the rest in system RAM.`, and a warning when less of it is on the GPU than usual, because another program or model is using GPU memory | | A dense model doesn't fit on the GPU (at startup and on `/model`) | `⚠ qwen3.8-27b is a dense model: all 27B parameters work on every token. It's capable, but slow here. Only about 61% of it fits in the 12 GB of GPU memory…`, with what to do instead | | The model that looks at images needs the memory | `Loading the vision model … to look at bug.png; qwen3.6-35b makes room and reloads afterwards`, then `◆ qwen3.6-35b was unloaded to make room…`, and the next step says `Reloading qwen3.6-35b into memory (it made room for the vision model …)` | | A tool needs the GPU to itself (an MCP server with `free_gpu`) | `◆ Unloaded qwen3.6-35b so the server can use the GPU; it reloads afterwards.` | @@ -421,6 +421,12 @@ While the model loads in the background at startup, the status bar says part of them in system RAM. Dense ones slow down a lot, to a few tokens a second, which is why they get the warning. +The `~` marks lcode's estimate of the model's memory, the same one `lcode setup` and `lcode doctor` +show. Ollama leaves out what stays mapped from the model's file in system RAM: `ollama ps` shows +qwen3.6-35b at 32K on a 12 GB GPU as 10.5 GB, 98% of it on the GPU, while ~12 GB of its experts are +in RAM. So when part of a model in `lcode models` is off the GPU, lcode counts all of it. For other +models it repeats what Ollama says. `lcode doctor` shows the same line for the model in memory. + ## Steer while it works A request can take minutes with a local model. You don't have to stop it to change course: type diff --git a/src/lcode/doctor.py b/src/lcode/doctor.py index cf75918..9bfb1ae 100644 --- a/src/lcode/doctor.py +++ b/src/lcode/doctor.py @@ -110,15 +110,9 @@ def line(label: str, value: str, good: bool | None = True) -> None: f"here (delete {limits.path()} to try again)", None, ) - loaded = [m for m in ollama.running() if m.get("name") == model or m.get("model") == model] - if loaded: - m = loaded[0] - line( - "Loaded", - f"{m.get('size_vram', 0) / 1e9:.1f} GB on GPU of {m.get('size', 0) / 1e9:.1f} GB, " - f"context {m.get('context_length', '?')}", - True, - ) + if where := residency.placement(ollama, model): # Ollama lists lcode-qwen3.6-35b as lcode-qwen3.6-35b:latest + where = residency.with_estimate(where, model, ctx) + line("Loaded", where.describe() + (f" · {format_tokens(where.context)} context" if where.context else "")) except cli.NotInstalled as e: line("Model", str(e), False) ok = False diff --git a/src/lcode/residency.py b/src/lcode/residency.py index b010777..4f3f965 100644 --- a/src/lcode/residency.py +++ b/src/lcode/residency.py @@ -12,17 +12,18 @@ import threading import time -from dataclasses import dataclass +from dataclasses import dataclass, replace from statistics import median from urllib.parse import urlsplit from lcode import catalog from lcode.cloud import is_cloud from lcode.config import format_tokens -from lcode.hardware import Hardware +from lcode.hardware import GIB, Hardware from lcode.ollama import OllamaError GB = 1e9 +ALL = 0.995 # this share of a model on the GPU counts as all of it, and 1 - ALL as none of it RELEASE_WAIT = 15.0 # seconds to wait for Ollama to give an unloaded model's GPU memory back SPLIT_WARNING = 0.95 # a dense model with less than this on the GPU runs much slower SHORTFALL = 0.25 # this much less on the GPU than expected: something else is using GPU memory @@ -36,24 +37,30 @@ @dataclass class Placement: - """A loaded model, from Ollama's /api/ps.""" + """A loaded model, from Ollama's /api/ps (see `with_estimate` for what that leaves out).""" name: str - size: float # bytes in memory + size: float # bytes in memory, as Ollama counts them vram: float # bytes of it on the GPU context: int = 0 # the context window it was loaded with + estimate: float = 0.0 # bytes the model takes by the catalog's estimate, when Ollama's count leaves part out + + @property + def total(self) -> float: + return max(self.size, self.estimate) @property def gpu_share(self) -> float: - return self.vram / self.size if self.size else 0.0 + return self.vram / self.total if self.total else 0.0 def describe(self) -> str: - """'22.3 GB, 44% on the GPU and the rest in system RAM'""" - size = f"{self.size / GB:.1f} GB" + """'6.6 GB, all on the GPU', or with the catalog's estimate: '~23 GB, 42% on the GPU and the rest in system + RAM'. The estimate is shown as `lcode setup` and `lcode doctor` show it.""" + size = f"~{self.estimate / GIB:.0f} GB" if self.estimate > self.size else f"{self.size / GB:.1f} GB" share = self.gpu_share - if share >= 0.995: + if share >= ALL: return f"{size}, all on the GPU" - if share <= 0.005: + if share <= 1 - ALL: return f"{size}, all in system RAM (on the CPU)" return f"{size}, {share:.0%} on the GPU and the rest in system RAM" @@ -117,6 +124,20 @@ def placement(ollama, model: str) -> Placement | None: return found[0] if found else None +def with_estimate(where: Placement, model: str, context: int) -> Placement: + """`where`, counting the whole model when part of it is off the GPU: by the catalog's estimate (models outside + the catalog stay as Ollama counts them). + + /api/ps counts the memory Ollama's runner allocates. The weights it keeps in system RAM aren't allocated: they + stay mapped from the model's file. So with Ollama 0.40, qwen3.6-35b at 32K on a 12 GB GPU shows as 10.5 GB, 98% + of it on the GPU, while ~12 GB of its experts are in RAM. A model all on the GPU is counted in full. + """ + spec = catalog.find(model) + if spec is None or where.vram >= ALL * where.size: + return where + return replace(where, estimate=spec.memory_gib(where.context or context) * GIB) + + def is_moe(ollama, model: str) -> bool | None: """Whether the model is a Mixture-of-Experts (None: unknown).""" spec = catalog.find(model) @@ -169,14 +190,17 @@ def load_report( ) -> str: """The line after a load: how long it took, how it fits, and a warning if it fits worse than it should.""" took = f" in {seconds:.0f}s" if seconds >= 1 else "" - line = f"{'Reloaded' if reload else 'Loaded'} {model}{took}" + (f": {where.describe()}." if where else ".") + line = f"{'Reloaded' if reload else 'Loaded'} {model}{took}" if where is None: - return line + return f"{line}." + where = with_estimate(where, model, context) + line += f": {where.describe()}." expected = expected_share(model, context, hardware) if expected is not None and where.gpu_share < expected - SHORTFALL: + fits = "all of it fits" if expected >= ALL else f"up to {expected:.0%} fits" line += ( - f" Less of it is on the GPU than usual ({expected:.0%}): another program or model is using GPU " - "memory, so it runs slower until that's closed." + f" Less of it is on the GPU than usual ({fits}): another program or model is using GPU memory, so it " + "runs slower until that's closed." ) elif dense and where.gpu_share < SPLIT_WARNING: line += " It's a dense model split between GPU and CPU, so expect it to be slow." diff --git a/tests/test_residency.py b/tests/test_residency.py index 59482ec..1c23865 100644 --- a/tests/test_residency.py +++ b/tests/test_residency.py @@ -21,6 +21,11 @@ def entry(name, size_gb, gpu_gb, context=0): return {"name": name, "size": size_gb * 1e9, "size_vram": gpu_gb * 1e9, "context_length": context} +# What Ollama 0.40.13 lists for lcode-qwen3.6-35b at 32K on the laptop: the ~12 GB of experts that stay mapped from +# the model's file in RAM aren't in its size, so 98% of it seems to be on the GPU. +OLLAMA_0_40 = {**entry("lcode-qwen3.6-35b:latest", 10.49, 10.29, 32768), "model": "lcode-qwen3.6-35b:latest"} + + def text(agent): return " ".join(output(agent).split()) @@ -31,6 +36,24 @@ def test_where_a_model_sits(): assert Placement("m", 6e9, 0).describe() == "6.0 GB, all in system RAM (on the CPU)" +def estimated(model, size_gb, gpu_gb, context=32768, fallback=32768): + return residency.with_estimate(Placement(model, size_gb * 1e9, gpu_gb * 1e9, context), model, fallback).describe() + + +def test_the_part_of_a_catalog_model_ollama_leaves_out_is_counted(): + assert estimated("lcode-qwen3.6-35b", 10.49, 10.29) == "~23 GB, 42% on the GPU and the rest in system RAM" + assert f"~{catalog.find('qwen3.6-35b').memory_gib(32768):.0f} GB" == "~23 GB" # as lcode setup and doctor say + assert estimated("qwen3.6-35b", 10.49, 10.29, context=262144) == "~28 GB, 35% on the GPU and the rest in system RAM" + assert estimated("qwen3.6-35b", 10.49, 10.29, context=0) == "~23 GB, 42% on the GPU and the rest in system RAM" + assert estimated("qwen3.5-4b", 1.2, 0) == "~5 GB, all in system RAM (on the CPU)" # no GPU: all of it is in RAM + + +def test_what_ollama_counts_stands_when_nothing_is_left_out_or_nothing_is_known(): + assert estimated("qwen3.5-9b", 8.5, 8.5, context=65536) == "8.5 GB, all on the GPU" # below the estimate + assert estimated("qwen3.6-35b", 26, 10) == "26.0 GB, 38% on the GPU and the rest in system RAM" # above it + assert estimated("custom:7b", 10.49, 10.29) == "10.5 GB, 98% on the GPU and the rest in system RAM" # no catalog + + def test_dense_models_that_dont_fit_get_a_warning(make_agent): ollama = make_agent().ollama warning = residency.dense_warning(ollama, "qwen3.8-27b", 32768, LAPTOP) @@ -54,29 +77,42 @@ def test_models_outside_the_catalog_are_asked_about(make_agent): def test_a_load_is_reported_once_with_how_it_fits(make_agent): - agent = make_agent([reply("hi"), reply("again")], model="qwen3.6-35b", context=32768) + agent = make_agent([reply("hi"), reply("again")], context=32768) agent.hardware = LAPTOP agent.ollama.loaded = [] def chat_stream(payload, on_open=None): agent.ollama.payloads.append(payload) - agent.ollama.loaded = [entry("qwen3.6-35b:latest", 22.3, 9.8, 32768)] + agent.ollama.loaded = [OLLAMA_0_40] yield from agent.ollama.scripts.pop(0) agent.ollama.chat_stream = chat_stream agent.run_turn("hello") - assert "◆ Loaded qwen3.6-35b: 22.3 GB, 44% on the GPU and the rest in system RAM." in text(agent) + assert "◆ Loaded lcode-qwen3.6-35b: ~23 GB, 42% on the GPU and the rest in system RAM." in text(agent) + assert "⚠" not in output(agent) # as much is on the GPU as usual agent.console.file.truncate(0) agent.run_turn("and again") assert "◆" not in output(agent) and "⚠" not in output(agent) # in memory: nothing to say -def test_less_on_the_gpu_than_usual_is_explained(make_agent): - report = residency.load_report("qwen3.6-35b", 14, Placement("m", 22.3e9, 2e9), 32768, LAPTOP, dense=False) - assert report.startswith("Loaded qwen3.6-35b in 14s: 22.3 GB, 9% on the GPU") - assert "another program or model is using GPU memory" in report - dense = residency.load_report("qwen3.8-27b", 40, Placement("m", 18e9, 11e9), 32768, LAPTOP, dense=True) - assert dense.endswith("It's a dense model split between GPU and CPU, so expect it to be slow.") +def report(model, size_gb, gpu_gb, context=32768, dense=False): + placement = Placement("m", size_gb * 1e9, gpu_gb * 1e9, context) + return residency.load_report(model, 14, placement, context, LAPTOP, dense=dense) + + +def test_less_on_the_gpu_than_usual_is_explained(): + # Another program holds GPU memory: Ollama counts 4.2 GB, 95% of it on the GPU, and leaves out the rest in RAM. + assert report("qwen3.6-35b", 4.2, 4.0) == ( + "Loaded qwen3.6-35b in 14s: ~23 GB, 16% on the GPU and the rest in system RAM. Less of it is on the GPU than " + "usual (up to 53% fits): another program or model is using GPU memory, so it runs slower until that's closed." + ) + assert report("qwen3.6-35b", 10.49, 10.29) == ( + "Loaded qwen3.6-35b in 14s: ~23 GB, 42% on the GPU and the rest in system RAM." # the usual 9-10 GB + ) + assert "(all of it fits)" in report("qwen3.5-9b", 7.7, 7.5, context=131072, dense=True) + split = "It's a dense model split between GPU and CPU, so expect it to be slow." + assert report("qwen3.8-27b", 18, 11, dense=True).endswith(split) + assert report("qwen3.8-27b", 11.2, 11, dense=True).endswith(split) # with the part in RAM left out, too def test_the_status_line_says_what_comes_before_the_first_token(make_agent): @@ -390,6 +426,18 @@ def test_doctor_says_nothing_about_ram_it_cant_tell(cli_here, machine): assert "RAM " not in cli_here() +def test_doctor_shows_how_the_loaded_model_fits(cli_here, machine, monkeypatch): + machine(available=25.0, ollama_private=0.0) + ollama = LocalOllama() + ollama.loaded = [OLLAMA_0_40] # as lcode-qwen3.6-35b:latest, while the model resolves to lcode-qwen3.6-35b + monkeypatch.setattr(cli, "Ollama", lambda host: ollama) + assert run_doctor() == 0 + assert "✓ Loaded ~23 GB, 42% on the GPU and the rest in system RAM · 32K context\n" in cli_here() + ollama.loaded = [entry("qwen3.5-9b", 6.6, 6.6, 131072)] # another model + assert run_doctor() == 0 + assert cli_here().count("Loaded") == 1 + + def test_setup_warns_about_ram_on_its_recommendation(cli_here, machine, monkeypatch): machine(available=22.0, ollama_private=0.0) monkeypatch.setattr(cli.Confirm, "ask", lambda *args, **kwargs: False)