Fix(agent): Improve tunnel preservation during agent upgrade when using tunnel. Housekeeping of shellhub-agent files in own directory

[ltan10]

What kind of change does this PR introduce?

• Bugfix
• New Feature
• Feature Improvment
• Refactoring
• Documentation
• Other, please describe:

Description:

• Organized go based agent config and key files into own dedicated directory in /etc/shellhub-agent/
  • Does not remove existing /etc/shellhub.key and /etc/shellhub-agent.env or /opt/shellhub/
  • Best migration procedure would be to copy shellhub.key to /etc/shellhub-agent/shellhub.key prior to performing installation/upgrade
• Modified default PRIVATE_KEY location value for podman, snap and docker installation in shell install script to /etc/shellhub-agent
• Restarts shellhub-agent service at end of installation step instead of ending shellhub-agent at start of installation procedure.
  • Reduces risk of removing device from shellhub tunnel network if agent installation/re-installation/upgrade process was performed over shellhub tunnel

Migration Guide *Optional*

The following guide also applies when migrating from runc-shellhub-agent to the native go-shellhub-agent.

The following steps are only needed if you want to retain the same device unique ID in ShellHub and avoid creating a new pending request.

If the migration steps are not performed, the device will be registered as a new device, and the pending request will need to be accepted.

1. Manually create /etc/shellhub-agent/ directory
2. Copy shellhub.key from /opt/shellhub-agent/shelhlub.key or /etc/shellhub.key to /etc/shellhub-agent/shellhub.key
3. Perform agent installation/upgrade steps as normal
4. Ensure installation/upgrade was successful and tunnel is active
5. Able to remove the following /opt/shellhub-agent/ and /etc/shellhub.key and /etc/shellhub-agent.env

[otavio]

The biggest problem I foresee here is migrating the field-deployed agents.

So even though I agree with the idea of this change, we need to consider how we will migrate the existing agents to work with this new approach or provide a backward-compatible way for them to keep working.

How do you foresee solving this issue?

[ltan10]

With all the testing. I've stumbled upon this myself.
From what I have seen, the migration is fine. as the install script during the migration from roofts to go based binary for the agents already leave the config.json behind and recreate the the new env config file.

The only persistent remaining file is just shellhub.key

Two methods for transition:

1. Just install/add/upgrade agent as normal using install script as per normal

• New shellhub.key gets regenerated in the new directory and device appears for pending approval.
• Requires user to manually remove existing accepted devices due to hostname clashes and accept the new pending devices
  or

2. Use existing shellhub.key on device by manually transfering it to /etc/shellhub-agent prior to executing install script.

• install script agent upgrade will just use existing key.

@otavio someone should be able to replicate my findings for confirmation.
I have been generally just been using option 1. And have only tested on the standalone deployment.
If someone can verify that i have changed the correct sections for the other installation methods and test, that would be great.

EDIT:
Should be clear that the above mentioned methods only works with prebuilt agent assets from shellhub releases, so a new release would have to be done for a smooth transition.
Users who pull the repo without the prebuild agent asset release will have to manually build and tweak their install procedure.

[ltan10]

@otavio
It later occured to me that my testing was performed using a physical connection and not a shellhub tunnel.

You are correct that users using the tunnel to upgrade their agent will have their tunnel ceremoniously terminated when migrating from the runc binary to go-native binary.

I tried sending the shell install script to background and including using nohup to ignore sighup. However still no luck as the child process executing the agent install did not appear to initiate.

This observation was the same for an standalone agent upgrade made through the tunnel for the following situations

• go-agent (ungrouped) -> go-agent (ungrouped)
• go-agent (grouped) -> go-agent (grouped)
• go-agent (ungrouped) -> go-agent (grouped)

With all tunnels terminated when users used shellhub tunnel to upgrade the agent

[ltan10]

Found a work around, can someone review please and test the containerised installation/upgrade.

Problem:
Basically found the native binary was affected by sighup and terminates, attempt to execute it using nohup in background did not work and did not provide verbosity to users.

Running the installation under systemd-run worked, but once again was not verbose, and increased complexity in managing the existing service once finished.

Solution:
Instead of stopping shellhub-agent.service at the start of installation, we restart/start the service at the end of installation.
This maintains an active tunnel session till the last moment after binary has been extracted and service is registered.
The session ends however the service is active and tunnel connection is resumed.

Changes:
Agent installation no longer stops the service at the start of installation.
Instead the agent service is restarted at end of install procedure, hence enabling the service does not start the service to prevent duplicate connection request at first installation.

[ltan10]

The biggest problem I foresee here is migrating the field-deployed agents.

So even though I agree with the idea of this change, we need to consider how we will migrate the existing agents to work with this new approach or provide a backward-compatible way for them to keep working.

How do you foresee solving this issue?

@otavio I noticed that the migration from non native shellhub-agent to the go-native shellhub-agent triggers the same issue of a new pending device join request due to the change in location of shellhub.key from /opt to `/etc/, especially for standalone installations.

I have provided a migration path for users who don't wish to deal with just removing old device and re-accepting the new device above. By moving the shellhub-agent service restart to end of installation, this allows users to upgrade the agent over a shellhub tunnel as the tunnel is not terminated when the service is stopped at the start of the installation process.

[ltan10]

/review

[github-actions]

@ltan10 You are not authorized to request an explicit review. If you believe this PR needs a new automated review round, please tag the @shellhub-io/admin team and a team member can trigger it.

[ltan10]

@shellhub-io/admin

[otavio]

/review

[github-actions]

Claude finished @otavio's task in 23s —— View job

---

Code Review Complete

Reviewed 3 files across code quality, security, testing, Go patterns, and architecture — no issues found. The code looks good as-is.

Review notes:

• Path changes (/etc/shellhub.key → /etc/shellhub-agent/shellhub.key, /etc/shellhub-agent.env → /etc/shellhub-agent/shellhub-agent.env) are consistent across all three files (Go installer, OCI config, shell script).
• The systemctl enable + systemctl restart split (replacing enable --now) correctly preserves tunnel connections during upgrades. restart on a loaded-but-not-running unit behaves like start, so fresh installs work too.
• Directory creation with os.MkdirAll before writing the env file is correct.
• No cross-repo impact found in the cloud repo — no references to the old agent paths exist there.
• No breaking changes to exported interfaces or pkg/ packages.

If you've addressed feedback and want a new review, tag @shellhub-io/admin and a team member can trigger it.

---

[ltan10]

Rebased branch. Proposed change completed.

@otavio any chance PR to be merged for next release?

[otavio]

@ltan10 There is still a conflict that needs to be resolved. Can you take a look?

Besides this, I am wondering what tests of the upgrade you have done?

[ltan10]

@otavio rebased to the master branch to resolve the conflicts. I've also updated the comments to make more sense.

As for the testing, I have only performed the STANDALONE installation method, in both an upgrade and fresh installation capacity. There is no issues with a fresh agent install.

For an upgrade context, I have tested the upgrade over an active remote ShellHub SSH tunnel multiple times, and was successful each of those times. However like with every remote configuration there is a chance that it may fail.

Every upgrade for runc-agent and go-agent always rewrites it's shellhub-agent.env as is normal behavior.

Observations for the tests are mentioned in previous comments. I'll try to summarize them here.

When upgrading from the runc-agent to "ungrouped" go-agent or "ungrouped" go-agent to this "grouped" version, the existing private key is not migrated. As a result, the new agent will generate a new key, and the device will be treated as a new device registration and await acceptance on the dashboard. Subsequently any existing shellhub.key and shellhub-agent.env from the existing install needs to be manually cleaned.

A migration guide is included in the PR description with steps to preserve the existing key and avoid re-registration.

I have NOT tested the containerized install/upgrade of this new agent. It would be great if someone on the team is able to verify their behavior.