ClusterCockpit ClusterBench

A continuous regression benchmarking framework for your HPC cluster which is:

• JSON based
• cluster-aware
• simple and lightweight
• reliable for stress testing

This framework is currently being used with NHR Benchmark collection for stress testing all the components within a HPC node.

Note: This framework does not support CI/CD pipeline based regression testing.

Requirements

Tool	Version	Reason
Golang	>= 1.25.0	Building ClusterBench
MachineState	>= 0.6.0	For capturing system settings and execution environment
Perl 5	>= 5.0	For collecting power,temperature and frequency data

All these tools should be available in the user's $PATH environment variable.

Build

• Build with:

make

• Clean up build files:

make clean

• Clean up build files and delete ./var directory + scheduler.db:

make distclean

• Make backup of your ./var directory + scheduler.db:

make backup

• Restore ./var directory + scheduler.db from the last backup:

make restore

Usage

./clusterbench [Command-Line-Arguments]

Command Line Arguments

Option	Argument	Default	Description
-h	—	—	Show help text.
-b	string	"benchmarks.json"	Filename/Filepath to your JSON benchmark file.
-m	bool	false	Enables monitoring mode for submitted benchmarks. Monitors failing/completion of the benchmarks.
-e	bool	false	Enables result extraction mode. Parses logs/measurements, extracts data, and writes to configured output (before benchmarking).
-u	bool	false	Enable update-only mode while startup. Checks for completition of previously submitted benchmarks, updates in database and exits. No benchmark submission

Examples

Launch ClusterBench with default benchmark file: benchmarks.json (if present).

./clusterbench

Launch ClusterBench with user-specifed benchmark file: benchmark-all.json. Will exit after submitting the benchmarks. No monitoring of the jobs.

./clusterbench -b benchmark-all.json

Launch ClusterBench to submit the benchmarks from benchmark-all.json and then monitor them untill every benchmark completes/fails.

./clusterbench -b benchmark-all.json -m

Launch ClusterBench to only update the scheduler.db database with the previously submitted benchmarks.

./clusterbench -u

Run result extraction on completed jobs (based on scheduler.db status) before submitting new ones:

./clusterbench -e -b benchmarks.json

Directory structure

clusterbench
│   README.md
|   config.json     //a JSON based config for clusterbench
|   scheduler.db    //a sqlite3 database for storing and tracking benchmarks
└───examples
|   // example benchmark JSON files and respective job file
└───internal
|   // golang code for clusterbench
└───scripts
|   // useful scripts for measurements and timing
└───var
    // directory for creating job file, submitting jobs and storing output

Config file

Example configuration in config.json:

{
    "retry-pending-on-startup": false,
    "max-queue-size": 1000,
    "multinode-permutation": false,
    "scheduler": {
        "type": "slurm",
        "active-states": [
            "idle",
            "alloc",
            "mix"
        ],
        "active-partitions": [
            "main",
            "backup",
            "preempt"
        ]
    }
}

Top-Level Options

Option	Type	Default	Description
retry-pending-on-startup	bool	false	All benchmarks marked as PENDING in scheduler.db will be queued and submitted on next startup.
max-queue-size	int	100	Maximum number of benchmarks to be submitted
multinode-combination	bool	false	Enables combination of nodes when #nodes > 1
scheduler	object		(see Scheduler below) configuration (backend,partitions, states, ...)

Scheduler Options

Currently only SLURM scheduler is supported.

Option	Type	Default	Description
type	string	"slurm"	Scheduler backend implementation.
active-states	string[]	["idle", "alloc"]	List of Slurm node states monitored or used for scheduling.
active-partitions	string[]	["main"]	Partitions monitored or used for scheduling.

Extraction Options

Configuration for the result extraction module (invoked with -e).

Option	Type	Default	Description
mode	string	"json"	Output format: "json", "parquet", or "db" (DuckDB).
extract-likwid	bool	true	Extract Likwid performance metrics from log files.
extract-CPU-measurements	bool	true	Extract CPU measurement CSVs (temperature, etc.).
extract-GPU-measurements	bool	true	Extract GPU measurement CSVs (power, temp, freq, etc.).
extract-benchmark-scores	object[]	[]	Rules for extracting benchmark scores (Regex or Script).

Score Extraction Configuration

Each item in extract-benchmark-scores can use a Regex or an external Script:

{
    "benchmark-name": "HPL",
    "extraction-script": "scripts/extract_hpl.sh", 
    "extraction-regex": "" 
}

Scripts should accept two arguments: <file_path> and <cluster_name>, and print the score to stdout.

Benchmark file

Since this is a vast section, we choose to explain this in the examples README. All the JSON objects and properties are explained in the README.

In the examples folder you will also find more detailed examples on how to write your "benchmark.json" and how to use different json objects.

more-examples -> In this folder you will find how we design/write the benchmark.json and submit benchmarks cluster-wide.

A brief overview of various examples of JSON objects you can find in examples folder:

1. benchmark-list
2. benchmark-whole-partition
3. benchmarks
4. gpu
5. measurements
6. node-list

Limitations

This framework is in early stages of work and has a lot of limitations:

• No range based iterator for running parameterized benchmarks.
• No compilation support. The user must provide the binary/executable.
• No CI/CD pipeline integration.

License — MIT

MIT License

Copyright (c) 2025 ClusterCockpit.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.