Benchmark for Optical Music Recognition of String Quartet Scores

This is the experiments repository for the paper:

A Dataset and Benchmark for Optical Music Recognition of String Quartet Scores
Dongmin Kim, Brian Liu, Jose J. Valero-Mas, Dasaem Jeong
Proceedings of the 27th International Society for Music Information Retrieval Conference (ISMIR), 2026

It holds the model training/evaluation code and configs used to run the OMR benchmarks reported in the paper. This is one component of a larger project; the recommended entry point for the full picture (dataset, preprocessing pipeline, symbolic format libraries, and baselines) is the umbrella repository: MALerLab/string-quartet-omr-benchmark.

This repository contains:

1. Training and inference scripts for the the following models:
   • PyTorch implementation of the Zeus model
   • Custom implementation of the Sheet Music Transformer (SMT)
2. PyTorch implementation of image augmentation methods proposed in the Mayer et al. (2024)

This repository does not contains:

1. Training and Test datasets — see Data access.
2. Preprocessing scripts for the datasets. For your reference, the preprocessing scripts are available in the MALerLab/omr-data-preprocessor

If you want to tinker with or build upon this code, please checkout the License and Citation sections for in-depth information.

Part of the String Quartet OMR Benchmark

This repository holds the model training and evaluation experiments for "A Dataset and Benchmark for Optical Music Recognition of String Quartet Scores" (MALer Lab, Sogang University).

• string-quartet-omr-benchmark: umbrella entry point for the paper and the full repository constellation.
• ossq-omr: the OSSQ-OMR dataset — MuseScore/MusicXML annotation sources and their revision history.
• omr-data-preprocessor: the preprocessing pipeline that builds every derived symbolic format from the dataset sources.
• sqomr: model training and evaluation experiments. (this repository)
• lmxe: the LMXE symbolic format library (derived from OMR-Research/lmx).

Setup

Clone this repository together with its submodules (modules/lmxe, modules/omrdp):

git clone --recursive git@github.com:MALerLab/sqomr.git
cd sqomr

If you already cloned without --recursive, fetch the submodules with:

git submodule update --init --recursive

This project uses uv for dependency management and targets the Python version pinned in .python-version (3.10):

uv sync

Project structure

    - sqomr/
        - config/
            - data/
                - <data config>.yaml
            - nn_params/
                - decoder_params/
                    - <decoder config>.yaml
                - encoder_params/
                    - <encoder config>.yaml
                - <nn config>.yaml
            - config.yaml
        - sqomr/ # Main codes are in here
            - nn/ # neural network related codes
                - decoders.py
                - encoders.py
                - layers.py # nn layers
                - utils.py
            - attention_utils.py # hook for attention visualization
            - data_utils.py # dataset classes and dataset maker classes
            - image_utils.py # image augmentation util functions
            - loss_utils.py # loss functions
            - model_zoo.py
            - render_utils.py # score rendering utilities
            - sampling_utils.py # sampling utilities for inference
            - train_utils.py # training utilities
            - trainer.py # trainer classes
            - vocab_utils.py # vocabulary classes
        - modules/ # external modules managed by git submodules
        - metadata/ # contains split, preprocess variations
            - <dataset name>/
                - train-segments.csv
                - valid-segments.csv
                - test-segments.csv
                - <encoding>-vocabulary.txt
        - notebooks/ # .ipynb files for analysis and visualization (TRACKED)
        - notebooks_local/ # personal .ipynb files (NOT TRACKED)
        - outputs/ # hydra stuffs, mainly not used (NOT TRACKED)
        - wandb/ # (NOT TRACKED)
            - <run id>/ # see train.generate_experiment_name for run_id 
                - checkpoints/
                    - <model ckpt>.pt
            - valid_out/
            - inferenced_outputs/
            - debug/
                - <debug runs>/
        - train.py
        - eval.py
        - inference.py
        - pyproject.toml # for dependency management
        - uv.lock # for dependency management

Models

Architecture is selected via the MODEL_TYPE variable (nn_params=$MODEL_TYPE, see Training), which picks an encoder/decoder pair from config/nn_params/:

• zeus — PyTorch implementation of the Zeus encoder/decoder.
• smt — custom implementation of the Sheet Music Transformer (SMT) encoder/decoder.
• smt-zeus / zeus-smt — hybrid configs pairing one model's encoder with the other's decoder (sqomr/model_zoo.py looks up the encoder/decoder class named in each config's encoder_params.name/decoder_params.name).

Data access

This repo does not vendor the dataset. Get the OSSQ-OMR dataset one of two ways:

• Download the prebuilt release from Zenodo and/or Hugging Face (links pending).
• Regenerate it from the source annotations in ossq-omr with the omr-data-preprocessor (omrdp, vendored here as modules/omrdp).

The data is located through the DATA_DIR env var — consumed by train_model.sh and passed through as the data.data_dir Hydra override. The data_dir field in each config/data/<name>.yaml ships as ??? (Hydra's mandatory-value marker), so a run that does not supply a path fails immediately with MissingMandatoryValue: Missing mandatory value: data.data_dir rather than silently reading from the wrong place. Either set DATA_DIR, override data.data_dir=/path/to/data on the command line, or fill the field in the config you use.

Per-split segment CSVs and vocabularies are tracked in this repo under metadata/<dataset name>/ (see Project structure); only the underlying page images are expected to live under DATA_DIR. Dataset caches (.pt files under each metadata dir's cache/) are built automatically on first use when data.on_memory=True; no separate pre-building step is required.

Training

train_model.sh wraps train.py with the Hydra config overrides used for the paper's experiments. The script does not set any defaults itself, so export the following environment variables before running it:

Variable	Hydra override	Purpose
DATA	data=$DATA	data config name under config/data/
DATA_DIR	data.data_dir=$DATA_DIR	root directory containing the dataset
SEQ_TYPE	data.sequence.type=$SEQ_TYPE	sequence encoding (e.g. krn, abc, lmxe)
SEQ_VARIANT	data.sequence.variant=$SEQ_VARIANT	encoding variant/tokenization
MODEL_TYPE	nn_params=$MODEL_TYPE	nn_params config name under config/nn_params/
LR	train_params.initial_lr=$LR (max_lr is set to ${LR}5)	learning rate
SCHEDULER	train_params.scheduler=$SCHEDULER	LR scheduler
BATCH_SIZE	train_params.batch_size=$BATCH_SIZE	training batch size
MAX_SEQ_LEN	train_params.max_sequence_length=$MAX_SEQ_LEN	max token sequence length
N_INFERENCE_LOG	inference_params.n_inference_log=$N_INFERENCE_LOG	number of systems logged at inference time
MAX_WORKERS	eval_params.max_workers=$MAX_WORKERS	worker count for eval metric computation
IMG_HEIGHT	nn_params.encoder_params.cnn_stack.input_height=$IMG_HEIGHT	input image height
SEED	general.seed=$SEED	random seed
DATA_CONF_PREFIX, DATA_ALIAS, SEG_LEVEL	only used to build wandb_config.notes	free-form labels recorded in the wandb run notes

Example:

DATA=ossq_v2.4.1_256_fold0 DATA_DIR=/path/to/data SEQ_TYPE=abc SEQ_VARIANT=abcbpe256 \
MODEL_TYPE=zeus LR=0.0001 SCHEDULER=cosinelr BATCH_SIZE=64 MAX_SEQ_LEN=640 \
N_INFERENCE_LOG=4 MAX_WORKERS=8 IMG_HEIGHT=256 SEED=0 \
DATA_CONF_PREFIX=ossq_v2.4.1 DATA_ALIAS=ossq SEG_LEVEL=systemwise \
bash train_model.sh

Evaluation

• eval.sh runs eval.py (OMR-NED evaluation for the Zeus/SMT models in this repo) on the test_synth and test_scanned splits of a single wandb run. Edit EXP_DIR at the top of the script to point at the run you want to evaluate before launching it.
• inference.sh runs inference.py for ad-hoc inference on a dataset TSV outside this repo. Set the WANDB_DIR (wandb run dir) and DATA_TSV (segments TSV) environment variables before running; the script fails fast if either is unset.

The comparative evaluation of the LEGATO model as an external baseline, along with its full development history and results, is preserved on the ossq-v2.4.0-legato branch, for readers who want to dig into that work.

Analysis tools

verify_ckpt.py checks that old wandb runs' checkpoints load cleanly (reports missing/unexpected/shape-mismatched keys) into the current model code.

Citation

If you use this benchmark, please cite:

@inproceedings{Kim2026sqomrbench,
    title     = {A Dataset and Benchmark for Optical Music Recognition of String Quartet Scores},
    author    = {Dongmin Kim and Brian Liu and Jose J. Valero-Mas and Dasaem Jeong},
    year      = 2026,
    booktitle = {Proceedings of the 27th International Society for Music Information Retrieval Conference (ISMIR)},
}

Depending on which components you use, please also cite:

If you use LMXE (the LMX linearizer/format, see modules/lmxe), the PyTorch implementation of the Zeus model, or the Zeus baseline numbers, please cite:

@inproceedings{mayer2024lmx,
    title     = {Practical End-to-End Optical Music Recognition for Pianoform Music},
    author    = {Mayer, Ji{\v{r}}{\'\i} and Straka, Milan and Haji{\v{c}} jr., Jan and Pecina, Pavel},
    year      = 2024,
    booktitle = {18th International Conference on Document Analysis and Recognition (ICDAR 2024)},
    pages     = {55--73},
    doi       = {10.1007/978-3-031-70552-6_4},
}

If you use the OSSQ-OMR dataset, please also cite the OpenScore String Quartets corpus it is built from:

@inproceedings{gotham_openscore_2023,
    title     = {The “{OpenScore} {String} {Quartet}” {Corpus}},
    author    = {Mark R. H. Gotham and Maureen Redbond and Bruno Bower and Peter Jonas},
    year      = 2023,
    booktitle = {Proceedings of the 10th {International} {Conference} on {Digital} {Libraries} for {Musicology}},
    pages     = {49--57},
    publisher = {ACM},
    address   = {Milan Italy},
    copyright = {All rights reserved},
    isbn      = {9798400708336},
    language  = {en},
    urldate   = {2023-11-16},
    month     = nov,
    doi       = {10.1145/3625135.3625155},
    url       = {https://dl.acm.org/doi/10.1145/3625135.3625155},
}

If you use the SMT baseline numbers, please cite:

@inproceedings{riosvila2024smt,
    title     = {Sheet Music Transformer: End-To-End Optical Music Recognition Beyond Monophonic Transcription},
    author    = {Ríos-Vila, Antonio and Calvo-Zaragoza, Jorge and Paquet, Thierry},
    year      = 2024,
    booktitle = {18th International Conference on Document Analysis and Recognition (ICDAR 2024)},
    pages     = {20--37},
    doi       = {10.1007/978-3-031-70552-6_2},
}

If you use the LEGATO baseline (its comparative evaluation is documented in Evaluation) or its baseline numbers, please also cite:

@inproceedings{yang2026legato,
    title     = {LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR},
    author    = {Guang Yang and Victoria Ebert and Nazif Tamer and Brian Siyuan Zheng and Luiza Pozzobon and Noah A. Smith},
    year      = 2026,
    booktitle = {International Conference on Learning Representations (ICLR)},
    url       = {https://arxiv.org/abs/2506.19065},
}

Licenses

Source code in this repository is available under the MIT license.

Vendored submodules carry their own licenses (see Part of the String Quartet OMR Benchmark).