Private AI Assistant with Tool Use

A locally-run AI assistant (via Ollama) that can chat, search your documents (RAG), read & fill PDF forms, grow its own skills, and edit code in your local repos under human approval — with no data ever leaving your machine. No cloud APIs, no API keys, fully private.

Status: chat, agent loop, RAG, PDF tools, model management, persisted chat history, a file-based skills system, and a sandboxed coding agent are all working and tested, behind a React frontend backed by a FastAPI API. See the roadmap for what's next.

Built from scratch to understand how modern AI agents actually work under the hood: the tool-calling loop, retrieval-augmented generation, and function dispatch — the same patterns behind ChatGPT and Claude, running entirely on local open-source models.

What it does

• 💬 Chat with a local LLM through a clean React interface
• 🔧 Uses tools — the model decides when to call functions, and the agent runs them and feeds results back (real function-calling, not prompt hacks)
• 📄 Reads & fills PDFs — extracts text and fills interactive form fields from plain-English instructions
• 📚 Chats with your documents — upload PDFs and ask questions; answers are grounded in the content via semantic search (RAG)
• 🧩 Grows its own skills — reusable capabilities defined as files (skills/<name>/), editable from the browser, and the model can even write new instruction-based skills for itself on request
• 🔀 Manages models — switch between any installed, tool-capable Ollama model mid-conversation, or pull new ones straight from the browser with live progress (and remove them again) — no terminal required
• 🕘 Remembers past chats — every conversation is persisted and listed in a short history you can revisit or delete, not lost the moment you start a new one
• 🗑️ Un-attaches documents — remove an uploaded file and its indexed chunks together, from the same sidebar you uploaded it in
• 🛠️ Writes code in your own repos — describe a change on the Coding page, and a second, sandboxed agent plans it, edits files, and runs your tests inside a throwaway git worktree; you watch every step stream in live and review the diff before Approve lands it on your working branch (or Discard throws it away, untouched)

How it works

        React UI (web/)
             │  HTTP (/api/*)
        ┌────▼─────┐
        │ FastAPI  │   server.py — routing, active conversation, model choice
        │ server   │
        └────┬─────┘
             │
        ┌────▼─────┐        ┌──────────────┐
        │  Agent   │◄──────►│ Ollama (LLM) │   local model, tool-calling
        │  loop    │        └──────────────┘
        └────┬─────┘
   tool call │ result
        ┌────▼───────────────────────────┐
        │  Tools                          │
        │   • read_pdf / fill_pdf         │  pdfplumber + pypdf
        │   • search_documents (RAG)      │  ChromaDB vector search
        │   • skill__<name>               │  skills/<name>/ — no code change
        └─────────────────────────────────┘

The core is a tool-calling loop: the model receives the conversation plus a set of tool schemas (built-in tools + every discovered skill), decides whether to call one, and the agent executes it and returns the result — repeating until the model produces a final answer. Adding a capability is either writing a function and registering it, or dropping a skill file on disk — no restart required.

The coding agent is a second, separate loop, reachable only from the Coding page — never a chat tool:

   Coding page (web/)
        │  HTTP (/api/coding/*), steps stream live over SSE
   ┌────▼─────┐        ┌──────────────┐
   │  Coding  │◄──────►│ Ollama (LLM) │   same local model, tool-calling
   │  agent   │        └──────────────┘
   │  loop    │  coding_agent.py — its own _execute_coding_tool() dispatch
   └────┬─────┘
tool call│ result — every step appended to runs/<id>.json
   ┌────▼─────────────────────────────────────┐
   │  git worktree (scratch branch off HEAD)   │
   │   • list_files / read_file / write_file   │  confined to the
   │   • run_tests (configured test command)   │  worktree root
   └────────────────┬───────────────────────────┘
                     │ diff
              ┌──────▼───────┐
              │ human review │  Approve → merges to the working branch
              │ (Coding page)│  Discard → worktree removed, repo untouched
              └──────────────┘

The worktree is the sandbox, the diff, and the undo — all three. The agent only edits files and runs tests inside that disposable checkout; nothing touches the real branch until a human reads the diff and clicks Approve. Discard removes the worktree, and the real repo was never touched.

Tech stack

Area	Choice
Language	Python 3.11+ (backend), TypeScript (frontend)
Model serving	Ollama (qwen2.5 for chat, nomic-embed-text for embeddings)
Backend	FastAPI + uvicorn
Frontend	React + Vite + Tailwind (web/)
PDF	pdfplumber (read) · pypdf (fill AcroForm fields)
Vector store	ChromaDB
Skills	file-based (skill.yaml + prompt.md or run.py), no sandbox — see skills/README.md
Conversation history	file-based (conversations/<id>.json), same pattern as skills — no database
Coding agent	separate loop (coding_agent.py), isolated in a git worktree per run — see Engineering notes
Tests / lint	pytest (unit + live e2e) · ruff · oxlint

Quickstart

# 1. Install Ollama and pull the models
ollama serve
ollama pull qwen2.5 && ollama pull nomic-embed-text

# 2. Set up the backend
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

# 3. Build the frontend
cd web && npm install && npm run build && cd ..

# 4. Run (one process, serves the API and the built UI)
uvicorn server:app --port 8000

Then open http://localhost:8000, drop a PDF into the sidebar, and start asking questions. For frontend development, run cd web && npm run dev instead of the build step — it proxies API calls to :8000 and hot-reloads.

Engineering notes

A few decisions I made deliberately, and why:

• One choke point for the model. Every call to Ollama goes through a single thin client module. This keeps the agent and tools decoupled from the SDK and makes the whole system unit-testable without a running model — the test suite mocks that one seam. The frontend mirrors this with a single API client module (web/src/api.ts).
• Native function-calling over prompt parsing. Tool requests are read from the model's structured tool_calls, not scraped from text — more robust, and it shapes a clean conversation-history format.
• Retrieval and ingestion are separate flows that share one vector store, with a bounded, guard-railed agent loop that can't run away.
• Skills are files, not code changes. A skill is skill.yaml plus either prompt.md (instructions the model follows) or run.py (code it executes, no sandbox). The model's own create_skill tool can only ever write instructions — never code — by construction, not convention: this closes off a prompt-injection-to-code-execution path, since content the model reads (a document, a search result) can at worst talk it into adding a weird instruction to the conversation it already came from, never into writing runnable Python. See skills/README.md for the full reasoning.
• The frontend never owns assistant state. Conversation history, the active model, and skills all live server-side; the UI re-fetches after every mutation instead of keeping a parallel copy.
• Isolation replaces prohibition for the coding agent. Every other tool in this app is in-process with no sandbox because the chat model can never run code it wrote (create_skill structurally can't emit a run.py). The coding agent deliberately crosses that line, so instead: it is a separate loop with its own smaller tool set, started only by an explicit human action (never a chat tool), and it can only edit files and run tests inside a throwaway git worktree on a scratch branch. Approving copies that diff onto the real branch; discarding removes the worktree — the real branch is never touched either way. See plan.md's Phase 16–18 decisions for the full reasoning.

Tests

pytest tests/ -v      # 181 tests, no live model required
pytest -m e2e -v       # live-model tests; skips cleanly if Ollama isn't running
ruff check .
cd web && npm run build && npm run lint

The unit suite covers the agent loop (including the runaway-loop guard), PDF read/fill (including the trap where pypdf silently "succeeds" on non-form PDFs), the ingest → search retrieval path, the skills registry, the conversation registry (persistence, title derivation), the coding-agent loop and its worktree lifecycle (path-confinement rejection, apply/discard against a real throwaway git repo), the run log store, and the FastAPI contract — all mocked or against real-but-disposable git repos, no Ollama required. The e2e suite drives the same tools, the API, model switching, and the skills system (including self-authoring) against a real running model; a live e2e run of the coding agent itself is not yet in that suite (see roadmap).

Roadmap

Working today

• Chat with a local LLM via Ollama
• Tool-calling agent loop with a runaway guard
• PDF reading + AcroForm filling
• Document RAG (ingest → embed → semantic search), with the ability to un-attach a document (deletes the file and its indexed chunks)
• React web UI with document upload, a tool-activity log, and inline model selection
• Runtime model selection across installed, tool-capable Ollama models, plus pulling and removing models from the browser (with live progress)
• Persisted, multi-conversation history — switch between past chats or start a new one without losing the last
• File-based skills system, including browser-based authoring and model self-authoring of instruction skills
• Sandboxed coding agent — a dedicated Coding page starts a second, confined agent loop that edits and tests a real local repo inside a throwaway git worktree, with a live streaming step log, a diff viewer, and human Approve/Discard before anything reaches a real branch

Building next

• Multi-agent coding review — writer/tester/security-reviewer roles layered on the same coding-agent loop and worktree, one shared run log
• Live e2e coverage for the coding agent against a real model and a fixture repo
• Streaming responses (token-by-token, via SSE) for chat replies
• More advanced chat tools — web search, spreadsheet/CSV analysis, calendar & email drafting
• OCR for scanned (non-interactive) PDFs