Repository navigation
Move ingesters to Elixir - #834
Open
KavanPrice wants to merge 20 commits into
Open
KavanPrice wants to merge 20 commits into
KavanPrice wants to merge 20 commits into
Conversation
Start the application that will replace the JS Sparkplug historian, and later the UNS ingester and UNS historian. It reads the variables the JS historian and the Helm chart use, refuses to start when one is missing or invalid, and starts the Factory+ client and an empty Sparkplug path. Each part is off unless its variable is true, so a side-by-side run can't double one by accident. Run the Elixir CI checks on it, listing projects by path.
Write line protocol to InfluxDB from a buffer, in a task, so the processes making points never wait for InfluxDB, and a failed write is retried rather than restarting the pod. Retry network errors, 5xx, 429 (honouring Retry-After), 408, and 401, 403 and 404, which mean the target is wrong rather than the data; drop other 4xx batches with a log. Cap the buffer at BUFFER_LIMIT points, dropping the oldest, and write what is left on shutdown. Start the historian's writer ahead of the Factory+ client, so a client restart keeps its buffer and it stops after the parts that feed it.
SparkplugB.Value took a signed integer's low bits for its width, so
an Int8 sent as 300 decoded as 44, and unsigned values weren't checked
at all. Keep such a value as the raw {field, value} tuple instead, as
for other values the library can't convert, so a consumer never sees
a wrong value. Signed types still accept the type's own width and sign
extension to 32 or 64 bits, which is what encoders send.
Route each Sparkplug message to a process for its edge node, which decodes it, keeps the node's births in SparkplugB.Host, and hands the resolved metrics, with each one's Factory+ instance and schema metadata, to the consumers. Ask devices with no known birth to rebirth through Command Escalation, in a task, and never ask nodes, as the JS historian does. A node's process that crashes loses only that node's births: the next message starts a new one, which asks for a rebirth, and other nodes carry on. The metadata follows the JS historian's rules, quirks included, so the tags it leads to will match.
Add Ingest.StallMonitor, which replaces the JS services' stall and connection watchdogs. Where those exit the process, losing the write buffer, it restarts the Factory+ client with fresh credentials and a fresh connection, and the Sparkplug path with it, while the writers keep their buffers. If a check fires a third time in a row without recovering, it stops, and as a significant child it shuts the service down, so that Kubernetes restarts the pod, the writers writing what they hold first. The router now notes when it last had a message, for the stall check. LOG_LEVEL is applied when the application starts.
Fail the image build if a step in the dependency heredoc fails, as every other heredoc now does, and keep the Elixir images' build caches in GHCR, as releases can't read the GitHub Actions cache.
The chart now runs acs-ingest as the Sparkplug historian, so nothing deploys the JS one. Remove its source, its image build and its place in the root Makefile, which acs-ingest takes, and update the comments that pointed at its files.
Each ingester becomes a part: a subtree with its own Factory+ client, so its own principal, keytab and MQTT connection, its own stall monitor and its own processes, while the parts share the InfluxDB writers. A part restarts on its own; one that gives up still stops the service. The Sparkplug historian is the first part. Variables a part needs for itself now start with its name, and the chart runs one acs-ingest Deployment, mounting each enabled part's keytab.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces historian-sparkplug, uns-ingester-sparkplug and historian-uns with acs-ingest, one Elixir service that runs each of them as a part of one pod. Each part keeps its own principal and MQTT connection, and writes and publishes what the JS service did, except where the JS was wrong or crashed.
k5start.acs-ingestDeployment, built from the existing enable flags and mounting each part's existing keytab. Its image and log level are underingest.lib/ex-sparkplug; KDC discovery inlib/erl-kerlberos.