CLOS-6911: remove SysV runlevel links that shadow a real unit after the upgrade

[prilr]

What

Adds RemoveStaleSysvLinks, a Finalization-phase actor that removes SysV runlevel links left over from the source system where the target provides a real systemd unit for the same service.

Why

A package can ship both an init script and a systemd unit. The runlevel links chkconfig created on the source system belong to no package, so nothing removes them during the upgrade. On the target, systemd-sysv-generator turns such a link back into a unit that contests with the real one.

cl-MariaDB103-server is the case this was written for. It ships /etc/init.d/mysql on EL9 as well, the EL8 links survive, and MariaDB ends up started by mysqld_safe outside mariadb.service:

CGroup: /system.slice/mysql.service
  |- 1585 /usr/bin/sh /usr/bin/mysqld_safe
mariadb.service: Unit process 1585 (mysqld_safe) remains running
Failed to start MariaDB database server.

The init script is itself broken on EL9, where log_success_msg no longer exists.

leapp's own systemd state transition cannot see any of this. It works on units, and these are files under /etc/rc.d/rc*.d - the same blind spot as CLOS-4518, which was the .timer half of _SYSTEMCTL_CMD_OPTIONS = ['--type=service', ...]. This is the SysV half.

Two important aspects

It has to run before the target boots. On FirstBoot the generator has already read the links and started the service - PID 1585 above is early boot. The removal then only takes effect on the second boot, and a conversion whose finish stage fails never reaches it. FinalizationPhase runs in the upgrade initramfs against the mounted target root, which is where leapp's own SetSystemdServicesState applies unit states for the same reason.

"Is there a real unit?" must not be asked of a booted system. systemctl list-unit-files includes generator output - a unit generated from the link being removed - so the check is circular and answers yes for any SysV service with an init script. It also made the enable step actively harmful: cl-MariaDB103-server ships no mysql.service, so systemctl enable mysql.service reported "not a native service, redirecting to systemd-sysv-install" and called chkconfig, which re-creates the links.

Units are therefore read from files in /usr/lib/systemd/system and /etc/systemd/system - never /run/systemd, where the generator writes - and a unit provides its own name plus every name in its [Install] Alias. mariadb.service carries Alias=mysql.service, which is how the SysV name reaches the real unit, and that unit is what gets enabled.

Fixing only the phase would break it the other way: with no boot yet there is no generator output, no mysql.service appears, and the actor silently leaves the links alone in exactly the case it exists for.

Behavior

• Only removes links whose service has a real unit on the target. Where there is none, the init script is the only way that service runs, so the links are left alone.
• If a start link existed, the real unit is enabled first - the administrator's intent that the service starts at boot has to move to the unit, or the service silently stops starting.
• If enabling fails, the links are kept. Removing them after a failed enable would leave the service started by nothing at all.
• Reports what it removed; says nothing when there is nothing to do.

Scope

Narrower than it may look.

cl-MariaDB103-server   SHIPS /etc/init.d/mysql
cl-MariaDB106-server   no init script
cl-MariaDB1011-server  no init script
cl-MySQL80-server      no init script

Only cl-MariaDB103 ships one, so only hosts running it are affected. That is not a rare configuration: EL8's default mariadb module stream is 10.3 and MySQL Governor's auto follows it, so hosts arrive on cl-MariaDB103 without choosing it, and leapp maps the stream 1:1 across the upgrade.

Notes

• No upstream equivalent exists at the moment. The underlying problem is not CloudLinux-specific in principle - any distro with a package shipping both an init script and a unit can hit it, so it may be worth offering upstream.

[azheregelya]

Review body

Requesting changes for one blocking issue: the actor's systemctl enable and leapp's own set_systemd_services_states run in the same Finalization Main stage without a defined order, and in the configuration behind CLOS-6911 the disable usually runs last, leaving MariaDB with no start path at all after the upgrade. Details and the trace are inline on actor.py.

The fix is a one-token move to the After stage plus a test adjustment, both verified locally on this branch: the 28 existing tests pass before the change, and the AST-based phase test is the only one that has to change.

Everything else is green here: unit tests, flake8, and both CI checks on a2f8344.

---

Inline comment: repos/system_upgrade/cloudlinux/actors/removestalesysvlinks/actor.py line 39

Blocker: this actor and set_systemd_services_states run in the same Finalization Main stage with no defined order, and when this one runs first the fix inverts.

Leapp orders actors within a stage only by produce/consume edges (PhaseActors._sort); the tag filter returns tuple(set(...)), so two unrelated actors run in effectively arbitrary order. This actor consumes nothing and produces only Report, so nothing pins it relative to set_systemd_services_states, which applies SystemdServicesTasks in the same stage.

Trace for the configuration behind CLOS-6911 (source: mariadb.service disabled, MariaDB started only through S64mysql):

1. RPM transaction: the EL9 cl-MariaDB103-server %posttrans finds the mysql.service / mysqld.service alias links missing, recreates them and runs systemctl enable mariadb.service. That block is unconditional, outside the $1 checks.
2. ApplicationsPhase: TransitionSystemdServicesStates sees source disabled, presets disable/disable, target enabled, and emits to_disable=[mariadb.service].
3. FinalizationPhase, this actor first: enable is a no-op, every S/K link is unlinked, the report says the unit "keeps starting at boot".
4. FinalizationPhase, set_systemd_services_states next: systemctl disable mariadb.service removes the WantedBy link and both alias symlinks.
5. First boot: no S links, unit disabled, nothing starts MariaDB. Plesk's finish stage systemctl start now succeeds against a stopped server, so the ticket symptom disappears while the database is down after every later reboot.

In 20 constructions of the IPU workflow from this branch, this actor came before set_systemd_services_states 19 times. Step 4 is also the only thing that explains the alias being absent at first boot in the original incident, i.e. why a generated mysql.service existed at all.

Fix: run strictly after the Main stage, which is what the After stage exists for (same pattern as createfinishmarker and removeresumeservice on FirstBootPhaseTag.After):

    tags = (FinalizationPhaseTag.After, IPUWorkflowTag)

Producing SystemdServicesTasks(to_enable=[...]) instead would not fix it: setsystemdservicesstate.process() disables any unit that appears in both lists. The AST-based phase test needs a matching change, see the comment there.

---

Inline comment: repos/system_upgrade/cloudlinux/actors/removestalesysvlinks/actor.py line 23

This is the premise that hides the interaction described on the tags line, and it is not quite right. The transition does not see the rc links, but it does see mariadb.service flip from disabled on the source to enabled on the target (the package %posttrans enables it during the RPM transaction), and it then disables the unit in Finalization. That disable is what removed the alias symlinks in the original incident and let the generator build mysql.service from the init script.

Worth rewording to say exactly that, since it is the reason this actor has to run after set_systemd_services_states.

---

Inline comment: repos/system_upgrade/cloudlinux/actors/removestalesysvlinks/tests/test_removestalesysvlinks.py line 247

With the After-stage fix, _declared_tags() returns {'After', 'IPUWorkflowTag'} (the ast.Attribute branch yields attr), so this test fails for the correct change, and it would pass for a wrong-module import of the tag. The snactor
fixtures expose the loaded actor's real tag classes, so the stage can be pinned without parsing source. Verified on this branch: with the After tag both tests below pass; against the current actor the first one fails.

from leapp.snactor.fixture import current_actor_libraries, loaded_leapp_repository  # noqa: F401; pylint: disable=unused-import
from leapp.tags import FinalizationPhaseTag, FirstBootPhaseTag


def test_runs_after_finalization_main(current_actor_libraries):
    assert FinalizationPhaseTag.After in current_actor_libraries.tags


def test_does_not_run_on_first_boot(current_actor_libraries):
    assert FirstBootPhaseTag not in current_actor_libraries.tags

This also drops the function-local import ast / import os, which trip the repo's pylint (C0415 twice, W0404 reimport of os).

[prilr]

Review findings confirmed, and fixed in the follow-up.