Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
120 lines (91 loc) · 5.21 KB

CONTRIBUTING.md

File metadata and controls

120 lines (91 loc) · 5.21 KB
Raw

Contributing

Bring your own selector

Goal: better answers at the same communication budget, or cheaper selection at the same quality.

1. Add your selector

Write a callable that follows the selector contract and pass it to rclc.transfer(..., selector=my_selector).

For a contribution to the package, put the function in src/rclc/selectors/. Keep it small and leave sender state unchanged. Extend an existing end-to-end scenario to cover selection, delivery and receiver continuation; avoid a test per helper. Follow the development harness and run:

.venv/bin/python scripts/check.py

2. Run a small comparison

Run on CPU with the included two-question smoke set:

python examples/evaluate_selectors.py > comparison.json

CacheBack is always included. --selector defaults to a recent-positions baseline and also accepts qsnap, chunkkv or an importable module:callable:

PYTHONPATH=. python examples/evaluate_selectors.py \
    --selector my_selector:select --data questions.jsonl --budget 128 > comparison.json

Each JSONL line contains documents (one string per sender), question, and answers (acceptable short answers). For example:

{"documents": ["Cedar is owned by Maya Chen."], "question": "Who owns Cedar?", "answers": ["Maya Chen"]}

Both selectors receive the same cached sources, question and budget. Results include raw answers, normalized exact-match correctness, actual retained positions, payload bytes, transfer time and generation time.

  • Matching ignores case, punctuation and repeated whitespace. It does not judge paraphrases or use the paper's scorer.
  • Timings include cold calls, exclude model loading and source prefill, and are not benchmark latency claims.
  • The included cases are API smoke examples, not a quality panel. Use a pinned model revision and a held-out question set for comparisons.

Without --budget, the script uses relative r4; --ratio sets another ratio. The default is Qwen3-0.6B on CPU; set --model, --revision, --device cuda and --max-new-tokens as needed.

3. Compare on the paper's benchmarks

Use a separate checkout and environment for the frozen paper-v1 snapshot (replication guide). In that snapshot, docs/benchmarks.md describes the panels and docs/running.md explains runs and reports.

Benchmark Task Qwen baseline config on paper
FanOutQA Three senders deliver evidence to one receiver configs/qwen-fanoutqa-natural-dev50.toml
LongBench v2 Pass and rerank state through a four-chunk chain configs/qwen-longbench-coa-easy50-rerank-n50.toml
LongBench v2, bounded Same chain with a fixed position cap configs/qwen-longbench-coa-easy50-bounded-n50.toml

For example, in the paper checkout (branch paper), after installing its Qwen environment, reproduce the FanOutQA baseline with:

python -m rcc.data fetch fanoutqa-natural-dev50
python -m rcc.run.entry --config configs/qwen-fanoutqa-natural-dev50.toml \
    --bundle data/fanoutqa-natural-dev50

These registered runs use eight H100 80 GB GPUs. The CPU quickstart does not reproduce benchmark results.

Custom selector integration into the paper benchmarks is manual. The frozen runner does not accept the main branch's selector= callable. Start an experiment branch from paper-v1, integrate your method into the selected model's selection path, and register a distinct experimental arm and run. Preserve the frozen branch, tag and baseline configs.

Keep the evaluation panel, model, prompts, generation settings, seeds and scoring fixed between methods. Match relative rX or bounded budgets, including inherited positions in chains, and report actual retained positions and payload bytes. Keep tuning separate from the evaluation panels. The paper snapshot's docs/selector-dev.md describes the development panel; replaying its stored keep sets only reproduces existing selectors, not a new method.

4. Share the comparison

Include the selector code and experiment integration, exact revision and run commands, configs, hardware/runtime pins and raw reports. Compare answer accuracy, retained positions, payload bytes and selection time against CacheBack at the same budgets. If reporting completion latency, match hardware and concurrency and include selection and handoff costs. Label small runs and untested settings clearly. A useful contribution does not need to beat CacheBack.

Bring a new benchmark

New tasks, datasets and agent communication patterns are welcome, including cases where CacheBack struggles. A benchmark needs no new selector. Use the public API in a runnable experiment, or extend the research runner on a separate experiment branch; keep the paper snapshot frozen.

Include a reproducible data source and its license, fixed development/evaluation splits, sender/receiver setup, requests and scoring code. Provide one small end-to-end example and commands for the full evaluation. Compare CacheBack with a relevant baseline under matched conditions, reporting task quality, communication budgets and costs using the reporting guidance above. Explain what the benchmark tests that the existing panels do not.