FinMapReduce is a research framework for question answering over long financial documents with large language models. It was developed to study divide-and-conquer MapReduce processing alongside alternative long-document strategies, with shared benchmark and evaluation workflows. A web application provides an interactive interface over the pipelines.
Some financial questions can be answered from one passage, such as a reported figure. Others require evidence spread across sections of a filing to assess a trend or explain a company's position. Long filings therefore pose problems of evidence coverage, synthesis, noise, context size, cost, and latency. FinMapReduce explores these problems under consistent processing and evaluation workflows; it does not assume one architecture is best for every question.
The framework separates pipelines, dataset loaders, prompt sets, output formats, models, and evaluation. This modular design makes it easier to vary one part while holding the others consistent, whether testing a new dataset, comparing architectures, or running an ablation study to isolate a design choice.
document -> chunking -> independent map requests -> candidate intermediate results
-> relevance filtering -> reduce synthesis -> final answer
In the Map phase, each chunk is processed independently against the same question to extract information that might help answer it. These requests can run concurrently. Depending on the output format, relevance scores help discard low-value results before reduction, limiting noise and downstream token use. The Reduce phase combines retained information into a coherent answer across sections and can surface conflicting evidence; no single chunk has to contain the whole answer.
Chunk size and overlap balance local context and continuity against input size, irrelevant material, fragmented evidence at chunk boundaries, and cost. They are configuration choices, not fixed properties of the method. Metadata mode can add sentence-level references to document chunks and attempt to locate cited passages in the source text.
Truncation provides an architectural baseline: it selects text that fits a context budget and answers with one model request. This makes it possible to compare the added MapReduce processing with a simpler approach. Retrieval-augmented generation (RAG) and long-context inference offer further trade-offs in coverage, retrieval dependence, accuracy, cost, latency, and synthesis; the appropriate approach depends on the question and workload.
- FinanceBench and augmented FinQA workflows for command-line evaluation, with LLM-judge scoring and token and timing statistics.
- A web interface for the pipelines, with uploads of multiple PDF, TXT, or Markdown documents and metadata for each document.
- JSON, plain text, hybrid, and metadata MapReduce formats through the factory and web application. Metadata mode builds sentence-level evidence references and attempts to match passages to source chunks.
- OpenAI, OpenRouter, and local OpenAI-compatible model endpoints.
Python 3.10 or newer is required. From the repository root, install the package and its dependencies:
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e .
cp .env.example .envSet the relevant API key in .env for OpenAI or OpenRouter. A local OpenAI-compatible server needs no API key; configure its endpoint as described in Configuration. Dataset runs also require the corresponding data and document files.
Run the local server from the repository root:
uvicorn webapp.backend.main:app --host 0.0.0.0 --port 8000Open http://localhost:8000 for the UI or http://localhost:8000/docs for the API schema. To run the container instead, set up .env and run:
docker compose up -dThe web application accepts multiple files per question. Its default PDF parser is the lightweight pypdf path; other parsers can require optional system or Python dependencies. See the web application guide.
Run a small FinanceBench evaluation from the repository root:
python main_async.py --dataset financebench --approach mapreduce --format_type hybrid --num_samples 10 --pdf_parser pypdfFor FinQA, provide the dataset JSON and the directory containing its document Markdown files when they differ from the CLI defaults:
python main_async.py --dataset finqa --approach mapreduce --format_type hybrid --data-path data/finqa/finqa_subset_test.json --doc_dir ../edgartools_finqa --num_samples 10The CLI accepts json, hybrid, and plain_text for --format_type. Metadata mode is available in the web application and programmatic factory, but is not exposed by this CLI argument. See Examples for truncation and provider commands.
MapReduce formatters determine map extraction, filtering, reduce prompts, and final parsing. JSONFormatter uses structured results; PlainTextFormatter uses text results; HybridFormatter combines text map results with a structured reduce result. MetadataFormatter adds evidence references and passage-position resolution. Truncation uses TruncationFormatter and supports start, end, and smart strategies.
The CLI and web application support openai, openrouter, and local. OpenAI and OpenRouter require their respective keys. The local provider calls an already running OpenAI-compatible endpoint; the web application's URL comes from LOCAL_BASE_URL. The CLI uses the LLM client's localhost default. See Configuration for the different defaults.
FinanceBench pairs financial questions and reference answers with evidence in full source filings. Its open-source subset supports end-to-end document processing, even when the answer evidence is localized. The repository also supports an augmented FinQA workflow: questions are paired with their full source reports rather than only the original benchmark's pre-selected context, combining information location with numerical reasoning. This augmentation is a project workflow, not an official FinQA release.
Evaluation was part of the framework's original purpose. Both pipelines use shared processing and LLM-judge evaluation so configurations can be compared. Dataset runs record token use and timing, use asynchronous requests, and write results under the dataset's results directory. Configurable prompts, models, formats, and rate limits support repeatable experiments. Model judgments and generated evidence should be reviewed against the source documents.
src/core/ Pipeline contracts, implementations, and factory
src/loaders/ FinanceBench, FinQA, and web document loaders
src/formatters/ MapReduce and truncation formatters
src/llm/ Async client and rate limiting
src/evaluation/ LLM-judge evaluation
src/utils/ Document processing and prompt loading
webapp/backend/ FastAPI application
webapp/frontend/ Browser interface
config/prompts/ Prompt sets and templates
docs/ Guides and interface reference
Long document processing makes multiple model calls and can incur provider costs. PDF extraction quality depends on the parser and document layout. Evidence matching relies on generated passages and text matching, so verify cited material before relying on an answer.