Perlmutter SLURM — Quick HOWTO

Four single-node scripts. Pick one (or two, for monitor + processor).

Script	Purpose
`slurm/monitor.slurm`	monitor stack only, on a normal exclusive compute node (`j_dpe` + exporter + Prometheus + Grafana)
`slurm/monitor-longrun.slurm`	the same monitor stack, but on a `workflow`-QOS long-running node — currently sized for 30 days
`slurm/processor.slurm`	one pipeline container, reports to a remote monitor
`slurm/allinone.slurm`	monitor and one pipeline on the same node

The monitor stack runs j_dpe (Monitor FE) + PrometheusExporter + Prometheus

Grafana. Pipeline nodes point at it via ERSAP_MONITOR_FE.

Port note: port 9000 is busy on Perlmutter. All scripts use port 19000 for the Monitor FE. j_dpe --port and --monitor-port must always match.

Script structure

slurm/
├── monitor.slurm           regular QOS, exclusive cpu node, up to 8 h
├── monitor-longrun.slurm   workflow QOS, cron constraint, up to 30 days
├── processor.slurm         pipeline container, points at a remote monitor
├── allinone.slurm          monitor stack + pipeline on one node
└── lib/
    └── monitor-stack.sh    shared bash body sourced by both monitor scripts

`slurm/lib/monitor-stack.sh`

monitor-stack.sh is not a script you run directly — it is a bash library sourced by monitor.slurm and monitor-longrun.slurm. It contains all the logic that both monitor variants share, organised into nine phases:

Phase	What it does
Log layout	Creates `logs/monitor-<JOB_ID>/` and defines paths for per-service log files, the info file, env file, and state file
Host discovery	Resolves the compute node's routable IP (`hostname -I`), builds `ERSAP_MONITOR_FE=<ip>%<port>_java`, expands `SLURM_JOB_NODELIST`
Shutdown handler	Installs `SIGTERM`/`SIGINT`/`SIGHUP` traps; on signal: sends SIGTERM to all child PIDs, waits up to `SHUTDOWN_TIMEOUT` seconds, then SIGKILLs stragglers, writes `STATE=stopped` to the state file
Readiness helpers	`port_listening <port>` — checks a TCP port is open; `http_ok <url>` — checks an HTTP endpoint returns 200; `wait_for <name> <pid> <timeout> <check>` — polls until the check passes or the process exits or times out
Pre-flight	Verifies all required executables, directories, and config files exist before starting anything; aborts cleanly if any are missing. Sets `PROM_DATA_DIR` to `/dev/shm/...` (tmpfs, for `PROM_STORAGE=tmpfs`) or `$PROM_HOME/data` (persistent disk, for `PROM_STORAGE=persistent`)
Commands	Assembles the four command arrays (`DPE_CMD`, `EXPORTER_CMD`, `PROM_CMD`, `GRAFANA_CMD`) from the caller's configuration variables. Adds `--storage.tsdb.retention.size=1GB` to Prometheus only when `PROM_STORAGE=persistent`
Startup	Starts the four services in dependency order (`j_dpe` → `PrometheusExporter` → `prometheus` → `grafana server`), waiting for each to pass its readiness check before starting the next. Verifies `ersap_prometheus_exporter_up == 1` before declaring success
Discovery files	Writes `monitor-info.txt` (human-readable: all endpoints, PIDs, commands, log paths, SSH tunnel command) and `monitor.env` (machine-readable: key=value pairs that processor nodes can `source`). Prints the readiness banner to stdout
Supervision loop	Polls all four PIDs every 5 s; if any exits unexpectedly, dumps its last 20 log lines and calls `shutdown 1`

How the monitor scripts use the library

Each monitor script is ~50 lines: SBATCH directives, a configuration block that sets all the required variables, and a single source call:

# from slurm/monitor.slurm
: "${ERSAP_HOME:=/global/homes/g/gurjyan/...}"
: "${MONITOR_PORT:=19000}"
: "${SESSION:=test}"
: "${PROM_STORAGE:=tmpfs}"     # short job: use RAM-backed storage
: "${MONITOR_TYPE:=regular}"   # written into monitor-info.txt
# ... other vars ...

# shellcheck source=lib/monitor-stack.sh
source "${SLURM_SUBMIT_DIR}/slurm/lib/monitor-stack.sh"

# from slurm/monitor-longrun.slurm — identical except:
: "${PROM_STORAGE:=persistent}"  # 30-day job: keep metrics on disk
: "${MONITOR_TYPE:=longrun}"

The only functional difference between the two scripts is PROM_STORAGE and the SBATCH resource request. Everything else — startup, supervision, shutdown, discovery files — is identical and lives once in the library.

Adding a new monitor variant

To create a new monitor variant (different QOS, time limit, or storage behaviour), copy either monitor script, change the #SBATCH directives and any configuration defaults, and keep the source line. The library handles the rest.

cp slurm/monitor.slurm slurm/monitor-myvariant.slurm
# edit #SBATCH lines and : "${...}" defaults as needed
# the source line stays identical

0. One-time setup (login node)

$HOME is a global GPFS filesystem, the same on the login node and all compute nodes, so binaries and configs installed here are immediately available inside any Slurm job.

Install Prometheus and Grafana binaries

# Prometheus
PROM_VER=2.53.0
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VER}/prometheus-${PROM_VER}.linux-amd64.tar.gz
tar xzf prometheus-${PROM_VER}.linux-amd64.tar.gz
mv prometheus-${PROM_VER}.linux-amd64 $HOME/prometheus
mkdir -p $HOME/prometheus/data

# Grafana
GRAF_VER=13.2.3
wget https://dl.grafana.com/oss/release/grafana-${GRAF_VER}.linux-amd64.tar.gz
tar xzf grafana-${GRAF_VER}.linux-amd64.tar.gz
mv grafana-${GRAF_VER} $HOME/grafana
mkdir -p $HOME/grafana/{data,logs,plugins}

Upgrading an existing install. Grafana 13 renamed the start command: the grafana-server binary is gone, replaced by the grafana server subcommand (the Slurm scripts already use the new form). Two things to know before the first start:

Back up $HOME/grafana/data/grafana.db first. Grafana migrates its SQLite schema forward on startup and the migration is one-way — a 13.x database cannot be opened by an older Grafana.

Move the old tree aside rather than untarring over it (mv $HOME/grafana $HOME/grafana.old), then re-run deploy.sh to reinstall conf/custom.ini, the datasource, and the dashboard. Copy grafana.old/data/grafana.db into the new data/ only if you want to keep saved dashboards and users; the provisioned ERSAP Overview dashboard is recreated from the repo either way.

Clone the repo and deploy configs

git clone https://github.com/JeffersonLab/ersap-java.git ~/ersap-java
bash ~/ersap-java/perlmutter-setup/deploy.sh

deploy.sh copies the Prometheus scrape config (perlmutter-setup/prometheus/ersap.yml), Grafana custom.ini, datasource, dashboard provisioning, and the ERSAP Overview dashboard into the correct $HOME locations. Re-run it any time you edit those files.

Edit the account line

Open each .slurm file and change #SBATCH --account=amsc016 to your NERSC allocation if it differs.

Prerequisites not in this repo

ERSAP_HOME — a pre-built ERSAP install (./gradlew deploy on your build machine). Required by slurm/monitor*.slurm and slurm/allinone.slurm.
podman-hpc image — the pipeline container image (default docker.io/gurjyan/pet-sro:v1). Required by slurm/processor.slurm and slurm/allinone.slurm. Pre-pull with podman-hpc pull <image> on a login node if you want to skip the pull step inside the job.

1. Monitor only

Two variants — pick based on how long you need the stack to run:

slurm/monitor.slurm — regular QOS, cpu constraint, exclusive compute node (64 CPUs), up to 8 h by default. Use for ad hoc testing and debugging.
slurm/monitor-longrun.slurm — workflow QOS, cron constraint, lightweight long-running node, currently sized for 30 days. Uses --dependency=singleton so re-submitting under the same job name won't start a second copy. Requires NERSC to have approved workflow QOS access for your account.

Submit

cd ~/ersap-java && mkdir -p logs

# long-running (preferred for production):
sbatch slurm/monitor-longrun.slurm

# short / debug:
sbatch --qos=debug --time=00:30:00 slurm/monitor.slurm

Note the JOB_ID printed by sbatch.

Wait for readiness

# watch the job stdout — the readiness banner prints after all four services
# (j_dpe, PrometheusExporter, Prometheus, Grafana) are confirmed listening:
tail -f logs/monitor-<JOB_ID>.out

The banner reads ERSAP monitor successfully started and is only printed after /metrics, /-/ready, /api/health all respond and ersap_prometheus_exporter_up == 1. A failure at any step aborts the job with a non-zero exit code.

Discover the node and endpoints

cat logs/monitor-<JOB_ID>/monitor-info.txt

monitor-info.txt records: Slurm job id, short hostname, FQDN, expanded node list, endpoints (j_dpe, exporter, Prometheus, Grafana), PIDs, launch commands, log paths, and the SSH tunnel command.

# machine-readable subset — source this on processor nodes or in other scripts:
source logs/monitor-<JOB_ID>/monitor.env
# exports: MONITOR_HOST, ERSAP_MONITOR_FE, MONITOR_PORT, PROM_PORT, GRAFANA_PORT

Connect from your laptop

The tunnel command is also printed in monitor-info.txt:

ssh -N \
  -L 3000:<monitor-node>:3000 \
  -L 9090:<monitor-node>:9090 \
  <user>@perlmutter.nersc.gov
# Grafana:    http://localhost:3000  (admin / changeme — change on first login)
# Prometheus: http://localhost:9090

Follow the logs

tail -f logs/monitor-<JOB_ID>/j_dpe.log
tail -f logs/monitor-<JOB_ID>/exporter.log
tail -f logs/monitor-<JOB_ID>/prometheus.log
tail -f logs/monitor-<JOB_ID>/grafana.log

Inspect the running node directly

NODE=$(squeue -h -j <JOB_ID> -o '%N')
ssh "$NODE" 'ss -tlnp | grep -E ":(19000|9095|9090|3000)"'
ssh "$NODE" 'ps -o pid,cmd -p $(pgrep -d, -u $USER -f "j_dpe|PrometheusExporter|prometheus|bin/grafana")'

Cancel

scancel <JOB_ID>

Clean cancellation triggers the batch script's SIGTERM trap, which stops all child processes gracefully, waits up to 20 s, then SIGKILLs stragglers.

2. Processor only (monitor already running)

slurm/processor.slurm runs the pipeline container on a separate compute node and points it at an already-running monitor stack.

Submit

cd ~/ersap-java && mkdir -p logs

# Option A — point at a running monitor job via its env file:
export MONITOR_ENV_FILE=$PWD/logs/monitor-<monitor-JOB_ID>/monitor.env
sbatch slurm/processor.slurm

# Option B — set the monitor address directly:
export ERSAP_MONITOR_FE='<monitor-ip>%19000_java'
sbatch slurm/processor.slurm

MONITOR_ENV_FILE is sourced by the script at startup; it exports ERSAP_MONITOR_FE (and the other monitor variables) so the pipeline DPE knows where to send its reports. ERSAP_MONITOR_FE set directly has the same effect and is useful when the monitor is not a job managed by this repo.

Follow and cancel

tail -f logs/processor-<JOB_ID>/pipeline.log
scancel <JOB_ID>     # stops the container cleanly via SIGTERM

3. Monitor + pipeline on one node

slurm/allinone.slurm runs the full monitor stack and one pipeline container on a single exclusive compute node. The job exits automatically when the pipeline finishes; the monitor is torn down as part of the same cleanup trap.

Submit

cd ~/ersap-java && mkdir -p logs
sbatch slurm/allinone.slurm

Readiness and logs

tail -f logs/allinone-<JOB_ID>.out                    # readiness banner
cat  logs/allinone-<JOB_ID>/monitor-info.txt          # tunnel command, endpoints
tail -f logs/allinone-<JOB_ID>/j_dpe.log
tail -f logs/allinone-<JOB_ID>/exporter.log
tail -f logs/allinone-<JOB_ID>/pipeline.log

The SSH tunnel command is the same as for the monitor-only case — substitute the node from monitor-info.txt.

4. Add an external `/metrics` endpoint to the monitor's Prometheus

Any component that exposes a Prometheus-format /metrics endpoint (custom exporter, third-party service, another pipeline) can be scraped by the Prometheus running inside the monitor job — no code changes to ERSAP required.

The file to edit

perlmutter-setup/prometheus/ersap.yml is the Prometheus configuration used on Perlmutter. It ships with one job that covers the ERSAP PrometheusExporter:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: ersap-monitor          # the ERSAP PrometheusExporter
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets: ["localhost:9095"]

Append an additional entry under scrape_configs for every external endpoint you want to scrape.

Minimal entry — only `job_name` and `targets` are required

  - job_name: my-exporter
    static_configs:
      - targets: ["nid001234:8000"]

metrics_path defaults to /metrics and scrape_interval inherits the global 15s, so nothing else is needed for a plain HTTP endpoint on any port.

Full annotated entry

  - job_name: my-exporter

    # Path Prometheus calls on each target (default: /metrics).
    metrics_path: /metrics

    # Per-job overrides; omit to inherit the global values.
    scrape_interval: 15s
    scrape_timeout: 10s

    # Static target list. Multiple groups can carry different labels.
    static_configs:
      - targets:
          - "nid001234:8000"      # Slurm compute node + port
          - "localhost:8000"      # or the same node as the monitor
        labels:
          pipeline: "mypipe"      # any key/value pairs added to every series
          user:     "gurjyan"

    # Only needed if the endpoint requires HTTP Basic Auth.
    # basic_auth:
    #   username: "user"
    #   password: "secret"

    # Only needed for HTTPS endpoints.
    # scheme: https
    # tls_config:
    #   insecure_skip_verify: true

job_name must be unique across all entries in the file.

Redeploy and reload — no monitor restart needed

# 1. copy the edited file into the running Prometheus data directory
bash ~/ersap-java/perlmutter-setup/deploy.sh

# 2. tell Prometheus to reload its config (no restart, no gap in data)
curl -X POST http://<monitor-node>:9090/-/reload

# 3. verify the new target appears as UP
# open http://<monitor-node>:9090/targets in a browser (via the SSH tunnel)

Viewing the metrics in Grafana

The new job's metrics are immediately queryable in Grafana against the existing Prometheus datasource. The ERSAP overview dashboard only shows ersap_* series. To add panels for your external metrics — JSON format, PromQL patterns, unit IDs, template variables — see docker/README.md § Grafana configuration.

Ephemeral hosts

If the exporter runs on a Slurm-allocated node whose hostname changes per job, hard-coding it in ersap.yml won't scale. Switch that job to file_sd_configs instead: write the target JSON to $HOME when the job starts and delete it on exit, then point ersap.yml at that file:

  - job_name: my-exporter
    file_sd_configs:
      - files: ["/global/homes/g/gurjyan/my-exporter-targets.json"]
        refresh_interval: 30s

The target file format:

[{"targets": ["nid001234:8000"], "labels": {"pipeline": "mypipe"}}]

Common overrides (export before `sbatch`)

Var	Default	Applies to	What it controls
`ERSAP_HOME`	`/global/homes/g/gurjyan/work/ersap_installation/ersap_home`	monitor, allinone	path to the pre-built ERSAP install
`MONITOR_PORT`	`19000`	monitor, allinone, processor	`j_dpe --port` and `--monitor-port`; must match on all nodes
`SESSION`	`test`	all	ERSAP session tag applied to DPE reports and user metrics
`IMAGE`	`docker.io/gurjyan/pet-sro:v1`	processor, allinone	`podman-hpc` container image for the pipeline
`DATA_DIR`	`/global/cfs/cdirs/amsc016/haidis/ersap-data`	processor, allinone	host directory mounted into the container as the data root
`SERVICES_FILE`	`'$ERSAP_USER_DATA/config/pet_services.yaml'` (container-resolved)	processor, allinone	services YAML path, resolved inside the container
`PIPELINE_PULL`	`1`	processor, allinone	set `0` to skip `podman-hpc pull` when the image is already cached
`MONITOR_ENV_FILE`	—	processor	path to `monitor.env` from a running monitor job; sets `ERSAP_MONITOR_FE` automatically

Troubleshooting

Symptom	Check
Job stays in PENDING	`squeue --start -j <JOB_ID>` for estimated start time; `sprio -j <JOB_ID>` for priority
Readiness banner never prints	`tail logs/monitor-<JOB_ID>.out` — the script logs each startup step; look for the first failure
`ersap_prometheus_exporter_up == 0`	The exporter can't reach the Monitor FE. Check `exporter.log`; verify `j_dpe` started on port `19000`
Prometheus target DOWN	`curl http://<node>:9095/metrics` from the monitor node to isolate network vs config
Grafana empty after tunnel	Confirm the tunnel is up (`ss -tlnp` locally); check the `$prefix` variable in the dashboard matches `--metric-prefix` (default `ersap`)
`workflow` QOS job rejected	NERSC must approve `workflow` QOS for your account; use `slurm/monitor.slurm` until then
Container fails to start	Check `pipeline.log`; re-run with `PIPELINE_PULL=1` if the image may be stale

Cheat sheet

# job status
squeue -u $USER
scontrol show job <JOB_ID>
scontrol show hostnames "$(squeue -h -j <JOB_ID> -o '%N')"
squeue --start -j <JOB_ID>
sprio -j <JOB_ID>
scancel <JOB_ID>

# discover monitor endpoints
cat logs/monitor-<JOB_ID>/monitor-info.txt
source logs/monitor-<JOB_ID>/monitor.env

# follow logs
tail -f logs/monitor-<JOB_ID>.out
tail -f logs/monitor-<JOB_ID>/j_dpe.log
tail -f logs/monitor-<JOB_ID>/exporter.log
tail -f logs/monitor-<JOB_ID>/prometheus.log
tail -f logs/monitor-<JOB_ID>/grafana.log

# inspect the running node
NODE=$(squeue -h -j <JOB_ID> -o '%N')
ssh "$NODE" 'ss -tlnp | grep -E ":(19000|9095|9090|3000)"'
ssh "$NODE" 'ps -o pid,cmd -p $(pgrep -d, -u $USER -f "j_dpe|PrometheusExporter|prometheus|bin/grafana")'

# SSH tunnel (also printed in monitor-info.txt)
ssh -N -L 3000:<monitor-node>:3000 -L 9090:<monitor-node>:9090 <user>@perlmutter.nersc.gov

# Prometheus hot-reload after editing ersap.yml
bash ~/ersap-java/perlmutter-setup/deploy.sh
curl -X POST http://<monitor-node>:9090/-/reload

Navigation Menu

Repository navigation

HOWTO-perlmutter.md

HOWTO-perlmutter.md

Perlmutter SLURM — Quick HOWTO

Script structure

`slurm/lib/monitor-stack.sh`

How the monitor scripts use the library

Adding a new monitor variant

0. One-time setup (login node)

Install Prometheus and Grafana binaries

Clone the repo and deploy configs

Edit the account line

Prerequisites not in this repo

1. Monitor only

Submit

Wait for readiness

Discover the node and endpoints

Connect from your laptop

Follow the logs

Inspect the running node directly

Cancel

2. Processor only (monitor already running)

Submit

Follow and cancel

3. Monitor + pipeline on one node

Submit

Readiness and logs

4. Add an external `/metrics` endpoint to the monitor's Prometheus

The file to edit

Minimal entry — only `job_name` and `targets` are required

Full annotated entry

Redeploy and reload — no monitor restart needed

Viewing the metrics in Grafana

Ephemeral hosts

Common overrides (export before `sbatch`)

Troubleshooting

Cheat sheet

FilesExpand file tree

HOWTO-perlmutter.md

Latest commit

History

HOWTO-perlmutter.md

File metadata and controls

Perlmutter SLURM — Quick HOWTO

Script structure

slurm/lib/monitor-stack.sh

How the monitor scripts use the library

Adding a new monitor variant

0. One-time setup (login node)

Install Prometheus and Grafana binaries

Clone the repo and deploy configs

Edit the account line

Prerequisites not in this repo

1. Monitor only

Submit

Wait for readiness

Discover the node and endpoints

Connect from your laptop

Follow the logs

Inspect the running node directly

Cancel

2. Processor only (monitor already running)

Submit

Follow and cancel

3. Monitor + pipeline on one node

Submit

Readiness and logs

4. Add an external /metrics endpoint to the monitor's Prometheus

The file to edit

Minimal entry — only job_name and targets are required

Full annotated entry

Redeploy and reload — no monitor restart needed

Viewing the metrics in Grafana

Ephemeral hosts

Common overrides (export before sbatch)

Troubleshooting

Cheat sheet

`slurm/lib/monitor-stack.sh`

4. Add an external `/metrics` endpoint to the monitor's Prometheus

Minimal entry — only `job_name` and `targets` are required

Common overrides (export before `sbatch`)