Skip to content

U1F30C/librarian

Repository files navigation

Librarian

A personal document search tool. Point it at a directory of PDFs, text files, and images; it scans, caches, extracts, and indexes them, then lets you search or browse from a CLI or a web UI.

It will:

  • Scan a directory for PDF, text (txt/md/json/html), or image (jpeg/png/webp) files
  • Cache extracted file contents by content hash, so re-indexing skips unchanged files instead of re-running extraction/OCR
  • Extract plain text using pdf-text-reader (PDFs) and tesseract.js OCR (images)
  • Index extracted text in an embedded SQLite database, searchable by hybrid keyword (FTS5/bm25)
    • semantic (vector) search
  • Let you search or browse the index through:
    • CLI — see docs/cli.md for the full command reference
    • Web UIpnpm run dev:web for a dev server, or pnpm run build:web + pnpm run start:web for a single-process production build

Getting started

pnpm install
pnpm run cli -- index ~/path/to/your/documents
pnpm run cli -- search "some query"

See docs/cli.md for every command and flag (index, search, list, browse, show, mv, prune, result filtering by format/path, etc).

For contributors / AI agents

CLAUDE.md documents the codebase architecture, dev commands, and known gotchas for anyone (human or Claude Code) working on this repo.

Inspiration

I wanted to search in a bunch of pdf files, quickly. I used to use pdfgrep but that was a bit slow even with the cache and the cache didn't last for long. I thought it'd be funnier to make my own tool rather than modifying pdfgrep to extend the cache lifespan.

History

This project went through a few embedded/hosted search engines (elasticlunr, flexsearch, minisearch, Meilisearch, Xapian) before landing on the current SQLite FTS5 + vector hybrid approach. See docs/history.md for the full story, and openspec/backlog.md for known bugs and what's planned next.

About

Index and search utilities for document discovery

Topics

Resources

License

Stars

3 stars

Watchers

1 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors