The interfaces below describe the current Python entry points. Import them from the repository root after installing the package. For HTTP request details, see Web application.
src.core.factory.PipelineFactory registers loaders by dataset name and builds pipelines.
create_pipeline(dataset, format_type="json", dataset_loader=None, **config) builds a MapReduce pipeline. create_truncation_pipeline(dataset, strategy="start", context_window=128000, buffer=2000, max_document_tokens=None, dataset_loader=None, **config) builds a truncation pipeline. register_dataset(name, loader_class) adds a loader class to the registry. See src/core/factory.py for type annotations and the complete signatures.
Registered datasets are financebench, finqa, and webapp. create_pipeline() accepts json, plain_text, hybrid, and metadata; it requires llm and prompts_dict in config. create_truncation_pipeline() accepts start, end, and smart, and also requires llm and prompts_dict. FinQA truncation additionally requires doc_dir. Either method calls the loader class's from_config(**config) unless dataset_loader is supplied. register_dataset() requires a DatasetLoader subclass.
BasePipeline coordinates dataset runs, concurrent requests, statistics, and evaluation. Applications use process_dataset_async(...) for dataset runs or process_single_qa_async(qa_pair, document_cache=None) for one question. Subclasses implement single-question processing and compile_statistics(qa_data).
MapReducePipeline loads chunks, runs map requests, filters results, runs reduce synthesis, and delegates parsing to its formatter. If the formatter sets supports_evidence_positions, the pipeline also resolves evidence against loaded chunks. TruncationPipeline loads full document text, truncates it to the requested context budget, and invokes one answer request via TruncationFormatter.
src.loaders.dataset_loader.DatasetLoader requires load_data(data_path, num_samples=None), load_document_chunks(qa_pair, chunk_size, chunk_overlap), load_full_document(qa_pair), get_document_identifier(qa_pair), get_results_directory(), get_dataset_name(), and the class method from_config(**config). It provides default get_map_question(qa_pair) and add_dataset_config(config) methods. Chunk loading returns chunks and token count; full-document loading returns text and token count.
src.formatters.output_formatter.OutputFormatter defines map invocation, map-result preprocessing, reduce-context formatting, reduce invocation, result parsing, and evaluation formatter selection. Concrete MapReduce implementations are JSONFormatter, PlainTextFormatter, HybridFormatter, and MetadataFormatter. The latter sets supports_evidence_positions = True and implements parse_evidences() to locate generated passages in source chunks. TruncationFormatter is a separate formatter for full-document truncation and one model call.
src.llm.async_llm_client.create_async_rate_limited_llm() returns an AsyncLLMClient:
create_async_rate_limited_llm(
model_name="gpt-4o-mini", temperature=0.0, max_tokens=8000,
provider="openai", api_key_env=None, api_key=None, base_url=None,
rate_limit_config=None, parse_json=False,
)RateLimitConfig accepts requests_per_minute, tokens_per_minute, request_burst_size, and optional token_burst_size. parse_json=True selects JSON response processing; otherwise the client returns raw model responses. OpenRouter uses its API endpoint, local uses an OpenAI-compatible endpoint, and OpenAI uses the normal OpenAI endpoint. base_url overrides the provider URL. The local provider supplies a placeholder key internally if no key is provided; OpenAI and OpenRouter require real credentials.
Import load_prompt_set from src.utils.document_processing. It reads named sets from config/prompts/prompt_config.yml and returns the prompt dictionary passed to pipeline creation.