Operation and Maintenance

Operation and maintenance

Service commands

Each role installs a different set of units. The standard systemd verbs apply to all of them:

sudo systemctl status axiom-border
sudo systemctl restart axiom-border
sudo systemctl is-active axiom-border
sudo systemctl is-enabled axiom-border

The units, depending on role: axiom-border, axiom-influxdb, containerd, oda-lite.

Where the logs live

This is the single most useful thing to know when something misbehaves, because the important detail is not in the journal:

Everything at once, which is often what you want:

sudo journalctl -u axiom-border -u axiom-influxdb -u containerd -u oda-lite \
  --since "5 minutes ago" --no-pager

Log levels

Axiom Border — the logger block of configuration.yaml. Set logLevel to DEBUG, INFO, WARN or ERROR, then restart the service. See Configuration.

oda-lite agent — /etc/oda-lite/oda-lite.conf, [agent] section: log_level and debug. Restart the service afterwards.

InfluxDB and containerd — the level comes from binary flags in the unit file. Changing it means editing the unit, and is not recommended.

Rotation and disk usage

Axiom Border’s own log rotates through the logger settings: maxSize (MB per file, default 20), maxBackups (default 20) and compress (gzip the rotated files).

The journald drop-in installed by the bundle is global — SystemMaxUse=500M and MaxFileSec=1day apply to the entire host journal, not only to Axiom Border. On a host sharing other workloads, adjust it or replace it with per-unit drop-ins.

journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d

To see what is consuming space:

du -sh /var/log/oda-lite /opt/axiom-border/logs /var/log/journal/

Backup and restore

Stop the service first, for a consistent snapshot:

sudo systemctl stop axiom-border
sudo tar -czf /path/backup-axiom-border-$(date +%F).tar.gz \
  /opt/axiom-border/config /opt/axiom-border/db.dat /var/lib/influxdb2
sudo systemctl start axiom-border

Restore is the inverse:

sudo systemctl stop axiom-border
sudo tar -xzf backup-axiom-border-<date>.tar.gz -C /
sudo systemctl start axiom-border

Offline maintenance

Refreshing the web and CVE vulnerability templates

The OT/ICS suite ships with the product and needs no maintenance. The web and CVE templates do, and on an air-gapped probe the refresh is manual: generate a snapshot of the upstream nuclei-templates tree on a machine with internet, transfer it, then swap it in.

# On a machine with internet
nuclei -update-templates -ut /tmp/nuclei-templates
tar -czf nuclei-templates-$(date +%F).tar.gz -C /tmp nuclei-templates

# On the probe
sudo systemctl stop axiom-border
sudo mv /opt/axiom-border/vulnscan-templates /opt/axiom-border/vulnscan-templates.old
sudo tar -xzf nuclei-templates-<date>.tar.gz -C /tmp
sudo mv /tmp/nuclei-templates /opt/axiom-border/vulnscan-templates
sudo systemctl start axiom-border

Keep .old until you have confirmed a few successful scans, then delete it. Templates are read when the service starts, so a restart is required after swapping them.

Loading AI images

AI images are not part of the bundle — they are distributed separately as OCI tarballs, and resources/iaimages/ ships empty as a placeholder.

sudo nerdctl --namespace axiom-border image load -i <image>.tar
sudo nerdctl --namespace axiom-border image ls

Upgrading

Extract the new bundle of the same role on the target and run upgrade.sh. It aborts if the bundle role does not match the installed role — changing role requires uninstall followed by install.

sudo ./upgrade.sh

What it does: verifies the previous installation and role match, stops the role’s services (leaving InfluxDB and containerd running, so data in flight is not disturbed), takes timestamped backups of the binary and db.dat, replaces the binary and updatable assets, starts the service, and health-checks it.

Preserved across upgrade: config/, db.dat, .influx-creds, parquet/, logs/, /var/lib/influxdb2/, resources/iaimages/ and the oda-lite configuration.

Replaced: the binary, MIBS/, resources/gojascripts/, vulnscan-templates/, the documentation and the lifecycle scripts.

Uninstalling

sudo /opt/axiom-border/uninstall.sh

It stops and disables the role’s services, removes only the units it created, removes the journald and sshd drop-ins, and asks whether to keep the data — InfluxDB data, db.dat, logs and configuration.yaml. Answer non-interactively with AXIOM_KEEP_DATA=true|false.

It removes shared binaries only where the manifest marks them as installed by this deployment. It does not remove nmap, rsyslog or network-manager even if it installed them — those are host infrastructure and are left to the operator. It does not revert the netplan switch by default.

Running it on a clean system is safe and idempotent: it reports there is nothing to uninstall and exits 0.

AXIOM_RESTORE_NETPLAN=1 warns about possible connectivity loss. In practice an established SSH session usually survives, since the TCP connection is not torn down and the IP is retained — but do not count on it over a link you cannot recover from out-of-band.

Troubleshooting by symptom

Quick triage

Detailed cases

The service will not start. Three usual causes: the binary lost its execute bit (chmod +x /opt/axiom-border/axiom-border), a required port is occupied, or the YAML is invalid. Validate the configuration directly:

python3 -c "import yaml; yaml.safe_load(open('/opt/axiom-border/config/configuration.yaml'))"

Login fails with the correct password. login.pass must be the SHA-256 digest, not the plaintext. Recalculate and restart:

echo -n "<your-password>" | sha256sum | awk '{print $1}'

A port is in use. Identify the holder, then either stop it or move the Axiom Border port:

sudo ss -tlnp | grep -E ':(8083|1883|1888|8086|9090|9091) '

Ports 8083, 1883 and 1888 move via apiPort, mqtt.broker.port and mqtt.broker.ws.port. Ports 9090 and 9091 move in /etc/oda-lite/oda-lite.conf.

Vulnerability scans return zero findings, suspiciously. The web template directory is probably empty or incomplete:

ls /opt/axiom-border/vulnscan-templates/ | head

It should list per-protocol directories such as http/, network/ and ssl/. CVE templates live under http/cves/, not at the root. Refresh them as described above. OT/ICS findings are unaffected, because that suite ships with the product.

Sniffing captures nothing. Check the interface exists (ip -o link show), that the service runs as root (systemctl show axiom-border -p User), and that there is actually traffic (sudo tcpdump -i <iface> -c 10).

The UI loads but every API call returns 401. The JWT is missing or expired. Clear the browser’s localStorage and log in again.

nmap probes hang or take very long. A large target range with poor reachability, a high timeout, or udp: true with many ports. Narrow the range, lower the timeout, disable UDP.

High memory usage. Check for an AI capability working a large dataset, an Influx bucket with no retention policy, or logLevel: DEBUG left on.

“Orphaned installation detected.” Bundle artefacts exist without an .install-manifest. Three ways out: AXIOM_CLEAN_RESIDUALS=1 sudo bash install.sh moves the leftovers aside, sudo bash upgrade.sh preserves data, sudo bash uninstall.sh cleans best-effort.

upgrade.sh exits with code 2 reporting a rollback. The new binary failed its health check and the old one is back in service. Diagnose with the journal, the application log and the install log, then retry with a corrected bundle.

The monitoring node floods the journal with “connection refused” to port 9090. The central node is not up, a firewall blocks it, or the URL is wrong. The agent keeps retrying and buffers metrics while it waits, so a short outage loses nothing — a long one eventually will.

Error executing 'netstat': exit status 1. Minimal Ubuntu Server images lack net-tools. Not blocking — the other plugins continue:

sudo apt-get install -y net-tools && sudo systemctl restart oda-lite

NetworkManager is active but interfaces show as unmanaged. The netplan renderer was not switched over. Re-run install.sh --switch-to-network-manager; it detects the current state and completes the switch without reinstalling NetworkManager.

/etc/oda-lite/oda-lite.conf.new contains literal ${VARIABLE} placeholders. The file is an unexpanded template. Do not copy it over the live configuration — the agent would not start. Delete it and re-run the upgrade with the current bundle.

scp or tar -xzf fail with Permission denied in /tmp. Some base images ship /tmp with wrong permissions. It should be drwxrwxrwt:

ls -ld /tmp
sudo chmod 1777 /tmp