Linux 8.1.7: percussioncms service stop does not stop process; starts before DB on reboot

[natechadwick-intsof]

Summary

On Linux 8.1.7, CMS is still managed with classic SysV / init.d scripts (not the newer native-systemd dual-ship path from main #962). Two field failures:

1. /etc/init.d/<service> stop (or service <name> stop) does not actually stop the Percussion CMS / Jetty process.
2. On reboot, the init.d service starts before the database is ready, so CMS fails to start cleanly; operator must start it manually after MySQL/MariaDB is up.

Fix the init.d / rxjetty.sh packaging used by 8.1.x — do not assume a systemd unit-only fix.

Environment (field)

• Product: Percussion CMS 8.1.7 (Java 8 LTS)
• Service model: /etc/init.d/… (e.g. percussioncms / Jetty-derived init script from install)
• OS: Linux (e.g. AWS Linux); even when systemd is the init, the product still uses init.d scripts (systemd may only wrap SysV if at all)
• Not: native percussion-cms.service dual-ship as the primary path on this release

Problem 1 — init.d stop does not stop CMS

Observed

• service <name> stop / /etc/init.d/<name> stop returns without leaving Jetty dead.
• JVM still in ps, ports still open.
• Blocks clean restarts and upgrades.

Where to look (8.1.x / Jetty service scripts)

Artifact	Role
/etc/init.d/<ServiceName>	Installed from Jetty start-helper template (rxjetty.sh lineage)
stop) branch	Typically start-stop-daemon -K -p "$JETTY_PID" … or kill on PID file
JETTY_PID / /var/run/…	Wrong/stale PID → stop thinks nothing is running or signals wrong process
StopJetty.sh	Alternate stop path; may disagree with init.d PID location
User / privileges	Script may re-run as install owner; root stop path may not match how start wrote the PID

Acceptance (stop)

• /etc/init.d/<name> stop and service <name> stop leave no CMS/Jetty JVM for that install.
• service <name> start after stop starts cleanly (no second JVM).
• service <name> restart works end-to-end.
• Stale/missing PID file: either recover via process match or fail loud; never exit 0 while Jetty still runs.
• Documented one-liner for support if force-kill is ever needed.

Problem 2 — init.d starts before database on reboot

Observed

• After reboot, CMS init script runs while MySQL/MariaDB is not accepting connections.
• Startup fails; site down until manual service <name> start later.

Root cause in product init headers

Shipped LSB headers on the Jetty init helper only declare filesystem + network, e.g.:

### BEGIN INIT INFO
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
### END INIT INFO

There is no dependency on MySQL/MariaDB (e.g. $remote_fs alone is not enough; need explicit DB facility or a start-time wait).

On SysV (and systemd-sysv-generator), that means CMS can run at a boot priority before the DB service finishes starting.

chkconfig style lines in the same family of scripts have also used early priorities historically (e.g. comments about 2345 20 80 / late start notes) — start order relative to mysqld must be fixed and verified on 8.1.7 packaging.

Required behavior

• Local DB: CMS init starts after DB is up (LSB Required-Start / Should-Start, chkconfig priority, or install-time registration that orders after mysqld/mariadb).
• Remote DB: wait-for-DB in the start path (retry JDBC/TCP until timeout) so reboot works without a local DB init script.
• Do not hard-fail install when no local MySQL package is present.

Proposed direction (init.d-centric)

1. Update LSB INIT INFO on the shipped init script template, e.g. Should-Start: mysqld mysql mariadb / distribution-appropriate names (document per RHEL/Amazon/Debian).
2. Raise start priority so CMS runs late in the boot sequence (after DB), e.g. chkconfig / update-rc.d priorities used by the installer.
3. Add optional wait_for_db in /etc/default/<name> + start function (host/port from repository props or env) — best for AWS RDS.
4. Same treatment for DTS init.d if co-hosted (optional sibling).

Acceptance (boot order)

• Cold reboot with local MySQL/MariaDB: CMS reaches running without manual start.
• Remote DB: wait loop (or clear docs) so first boot after reboot succeeds when DB is reachable within timeout.
• Installer/docs list how to set DB wait host/port for non-local DB.
• No hard dependency that breaks installs with remote-only DB and no local mysqld init script.

Workaround (field, init.d)

Stop (until fixed):

# as root or install owner
sudo /etc/init.d/<name> status   # if available
ps aux | grep -i jetty
# if stop left it up:
sudo kill -TERM <pid>
# or install StopJetty.sh as owner of the install tree

Boot order (local MySQL example — Amazon/RHEL-style):

• Ensure mysqld/mariadb is chkconfig on / enabled.
• Soften race: delay CMS start, e.g. edit init script or use a late @reboot only as emergency.
• Prefer fixing Required-Start / priorities in the product script rather than long sleep hacks.

Remote DB: small wrapper or edit start) to loop until mysqladmin ping / nc -z dbhost 3306 before invoking Jetty.

Related

• This issue is init.d / 8.1.7, not “systemd-only.”
• Longer-term native systemd dual-ship: Migrated: Update the Linux service scripts to use systemd services instead of init.d percussioncms#962 (main) — still relevant later; 8.1.7 field path remains init.d.
• Scripts (main tree names; apply fixes on percussioncms-java8 packaging): Jetty rxjetty.sh → /etc/init.d/<ServiceName>, install-jetty-service.sh (init.d install), StopJetty.sh.

Priority

p2 / ops — stop that doesn’t stop blocks maintenance; boot-before-DB blocks availability after every reboot.

[natechadwick-intsof]

Correction from field: 8.1.7 on Linux still uses init.d scripts as the service path (not the main-line native systemd dual-ship as primary).

Issue body rewritten to center on:

• /etc/init.d/<name> stop / service <name> stop not killing Jetty
• LSB headers only Required-Start: $local_fs $network (no DB) → starts before MySQL/MariaDB on reboot

Systemd unit notes are secondary only if sysv-generator wraps the same init script.