Skip to content

[splunk_inputs] reload system stanzas on TA add and removal - #126

Draft
dmitryax wants to merge 3 commits into
mainfrom
fix-system-stanza-reload-on-ta-removal
Draft

dmitryax wants to merge 3 commits into
mainfrom
fix-system-stanza-reload-on-ta-removal

Conversation

@dmitryax

@dmitryax dmitryax commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Problem

splunkinputsreceiver reloads system-level stanzas (etc/system, not owned by any TA) only on an in-place TA modification. A full TA add or removal shifts stanza ownership between a TA and the system layer just as much, but the reconcile loop missed both cases.

A known TA's deletion routes to pending[taDir] rather than the "" all-TAs sentinel, so allTAs is false. The TA then lands in the removed branch, which never set taChanged, so systemKey was never appended to changed and ReadSystemInputs was not re-run.

Concrete failure: system config defines [monitor:///var/log/syslog] and TA splunk_ta_x also defines it. At Start the system copy is excluded and the TA collects it. Delete splunk_ta_x at runtime: reconcile stops the TA receiver but does not reload system stanzas, so the stanza is collected by neither and silently stops being ingested until an unrelated later add/modify or system-conf event forces a reconcile.

Fix

Track ownership changes across the added, removed, and changed buckets (excluding systemKey itself) and reload systemKey on any TA transition. Guard the reload on systemKey being currently active: when all system stanzas were TA-owned, systemKey is not in active and the added loop already reloads it, so scheduling a changed reload for a key with no stanzas is avoided.

Found by a revmux review of #125.

Testing

Added reconcile subtests: TA removal and targeted TA change both reload active system stanzas; a targeted change with no active systemKey does not conjure one. Verified the tests fail on the pre-fix code.

Follow-ups

Two further findings from the same review are out of scope here and will be separate PRs: the systemKey sentinel leaking into public ReceiverRequest.BaseDir, and new TA receivers starting before the losing system owner is stopped.

Only an in-place TA modification set the flag that reloads system stanzas.
A full TA add or removal shifts stanza ownership too: a known TA's deletion
routes to pending[taDir] (not the "" all-TAs sentinel), so allTAs is false,
the TA lands in the removed branch, and systemKey was never appended to
changed. A stanza the TA had claimed then stays suppressed at the system
layer until some unrelated later event forces a reconcile.

Track ownership changes across the added, removed, and changed buckets and
reload systemKey when any TA transition occurs, but only when systemKey is
currently active so a reload is not scheduled for a key with no stanzas.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant