# Operation and Maintenance

## Operation and maintenance

## Service commands

Each role installs a different set of units. The standard systemd verbs apply to all of them:

```bash
sudo systemctl status axiom-border
sudo systemctl restart axiom-border
sudo systemctl is-active axiom-border
sudo systemctl is-enabled axiom-border
```

The units, depending on role: `axiom-border`, `axiom-influxdb`, `containerd`, `oda-lite`.

## Where the logs live

This is the single most useful thing to know when something misbehaves, because the important detail is
**not** in the journal:

| Service | Journal | Own file | Rotation |
| --- | --- | --- | --- |
| `axiom-border` | systemd events and early startup errors only — with the shipped default, very little | `/opt/axiom-border/logs/axiom-border.log` — **all the detail**: probes, MQTT, scans, rules | Automatic, tunable in `configuration.yaml` |
| `axiom-influxdb` | Everything: startup, crashes, slow queries, compactions | None | journald drop-in: 500 MB, 1 day per file |
| `containerd` | Everything: runtime, namespaces, OCI errors | None | journald drop-in |
| `oda-lite` | Startup, crashes and early warnings only — 3 to 5 lines in normal operation | `/var/log/oda-lite/oda-lite.log` — plugin activity, HTTP errors towards central, emitted metrics | Automatic: every 24 h or at 100 MB, 7 archives kept |
| Lifecycle scripts | — | `/var/log/axiom-border-install.log` — install, upgrade and uninstall appended together | **Not rotated**; clear it by hand if it grows |

{{% notice style="tip" title="The rule of thumb" icon="lightbulb" %}}
To diagnose **application logic**, read the file. To diagnose a **crash or a failure to start**, read
the journal. Looking for probe results in `journalctl` is the most common wasted hour.
{{% /notice %}}

Everything at once, which is often what you want:

```bash
sudo journalctl -u axiom-border -u axiom-influxdb -u containerd -u oda-lite \
  --since "5 minutes ago" --no-pager
```

## Log levels

**Axiom Border** — the `logger` block of `configuration.yaml`. Set `logLevel` to `DEBUG`, `INFO`, `WARN`
or `ERROR`, then restart the service. See [Configuration](../configuration/#logger).

**oda-lite agent** — `/etc/oda-lite/oda-lite.conf`, `[agent]` section: `log_level` and `debug`. Restart
the service afterwards.

**InfluxDB and containerd** — the level comes from binary flags in the unit file. Changing it means
editing the unit, and is not recommended.

{{% notice style="warning" title="Turn DEBUG back off" icon="triangle-exclamation" %}}
`DEBUG` on a busy probe generates enough volume to fill a disk. Raise it to reproduce a problem, then
put it back to `INFO`. Do not leave it on "just in case".
{{% /notice %}}

## Rotation and disk usage

Axiom Border's own log rotates through the `logger` settings: `maxSize` (MB per file, default 20),
`maxBackups` (default 20) and `compress` (gzip the rotated files).

The journald drop-in installed by the bundle is **global** — `SystemMaxUse=500M` and `MaxFileSec=1day`
apply to the entire host journal, not only to Axiom Border. On a host sharing other workloads, adjust it
or replace it with per-unit drop-ins.

```bash
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d
```

To see what is consuming space:

```bash
du -sh /var/log/oda-lite /opt/axiom-border/logs /var/log/journal/
```

## Backup and restore

| Resource | Path | Suggested frequency |
| --- | --- | --- |
| Configuration | `/opt/axiom-border/config/configuration.yaml` | On every change |
| Local state — rules, aliases, executions | `/opt/axiom-border/db.dat` | Daily |
| InfluxDB data — metrics, alarms | `/var/lib/influxdb2/` | Per retention policy |
| Logs | `/opt/axiom-border/logs/` | On demand |

Stop the service first, for a consistent snapshot:

```bash
sudo systemctl stop axiom-border
sudo tar -czf /path/backup-axiom-border-$(date +%F).tar.gz \
  /opt/axiom-border/config /opt/axiom-border/db.dat /var/lib/influxdb2
sudo systemctl start axiom-border
```

Restore is the inverse:

```bash
sudo systemctl stop axiom-border
sudo tar -xzf backup-axiom-border-<date>.tar.gz -C /
sudo systemctl start axiom-border
```

## Offline maintenance

### Refreshing the web and CVE vulnerability templates

The OT/ICS suite ships with the product and needs no maintenance. The web and CVE templates do, and on
an air-gapped probe the refresh is manual: generate a snapshot of the upstream `nuclei-templates` tree
on a machine with internet, transfer it, then swap it in.

```bash
# On a machine with internet
nuclei -update-templates -ut /tmp/nuclei-templates
tar -czf nuclei-templates-$(date +%F).tar.gz -C /tmp nuclei-templates

# On the probe
sudo systemctl stop axiom-border
sudo mv /opt/axiom-border/vulnscan-templates /opt/axiom-border/vulnscan-templates.old
sudo tar -xzf nuclei-templates-<date>.tar.gz -C /tmp
sudo mv /tmp/nuclei-templates /opt/axiom-border/vulnscan-templates
sudo systemctl start axiom-border
```

Keep `.old` until you have confirmed a few successful scans, then delete it. Templates are read when the
service starts, so a **restart is required** after swapping them.

### Loading AI images

AI images are **not** part of the bundle — they are distributed separately as OCI tarballs, and
`resources/iaimages/` ships empty as a placeholder.

```bash
sudo nerdctl --namespace axiom-border image load -i <image>.tar
sudo nerdctl --namespace axiom-border image ls
```

## Upgrading

Extract the **new bundle of the same role** on the target and run `upgrade.sh`. It aborts if the bundle
role does not match the installed role — changing role requires uninstall followed by install.

```bash
sudo ./upgrade.sh
```

| Flag | Meaning |
| --- | --- |
| `--force-conf-regen` | Regenerate the oda-lite configuration from the new template, saving the current one as `.bak-<timestamp>` |
| `-h`, `--help` | Print help and exit |

| Variable | Default | Meaning |
| --- | --- | --- |
| `HEALTHCHECK_RETRIES` | 12 | Attempts before declaring failure |
| `HEALTHCHECK_INTERVAL` | 5 | Seconds between attempts — 12 × 5 s gives a 60 s window |
| `AXIOM_API_PORT` | 8083 | API port used for the health check |

What it does: verifies the previous installation and role match, stops the role's services (**leaving
InfluxDB and containerd running**, so data in flight is not disturbed), takes timestamped backups of the
binary and `db.dat`, replaces the binary and updatable assets, starts the service, and health-checks it.

**Preserved across upgrade:** `config/`, `db.dat`, `.influx-creds`, `parquet/`, `logs/`,
`/var/lib/influxdb2/`, `resources/iaimages/` and the oda-lite configuration.

**Replaced:** the binary, `MIBS/`, `resources/gojascripts/`, `vulnscan-templates/`, the documentation
and the lifecycle scripts.

{{% notice style="tip" title="Failed upgrades roll back automatically" icon="lightbulb" %}}
If the health check fails all 12 attempts, `upgrade.sh` **restores the previous binary**, confirms the
old version answers with HTTP 200, and exits with code 2. The probe is left operational on the old
version and the backup is kept at `/opt/axiom-border/axiom-border.bak-<timestamp>`. An upgrade that
reports a rollback has not broken anything — diagnose, then retry with a corrected bundle.
{{% /notice %}}

{{% notice style="warning" title="Back up `db.dat` before upgrading" icon="triangle-exclamation" %}}
The local state file `db.dat` is **not migrated automatically**. If a new version changes its format, the
file may have to be regenerated, and that loses local state — rules, aliases and execution history.
Always take a copy before upgrading, and if the new version will not start cleanly, restore the copy and
revert the binary.
{{% /notice %}}

## Uninstalling

```bash
sudo /opt/axiom-border/uninstall.sh
```

It stops and disables the role's services, removes **only** the units it created, removes the journald
and `sshd` drop-ins, and asks whether to keep the data — InfluxDB data, `db.dat`, logs and
`configuration.yaml`. Answer non-interactively with `AXIOM_KEEP_DATA=true|false`.

It removes shared binaries only where the manifest marks them as installed by this deployment. It does
**not** remove `nmap`, `rsyslog` or `network-manager` even if it installed them — those are host
infrastructure and are left to the operator. It does **not** revert the netplan switch by default.

| Variable | Effect |
| --- | --- |
| `AXIOM_KEEP_DATA=true\|false` | Answers the data-retention question without prompting |
| `AXIOM_RESTORE_NETPLAN=1` | Restores `/etc/netplan/` from the backup, re-applies it, and switches back to `systemd-networkd` |
| `AXIOM_FORCE_CLEAN=1` | **Destructive.** Without a manifest, claims every detected artefact as its own — including global binaries that may belong to other software such as Docker or Kubernetes |

Running it on a clean system is safe and idempotent: it reports there is nothing to uninstall and exits 0.

{{% notice style="warning" title="AXIOM_FORCE_CLEAN can remove another product's binaries" icon="triangle-exclamation" %}}
Use it only after confirming nothing else on the host depends on containerd, runc, nerdctl or the CNI
plugins. Without the flag, a manifest-less uninstall stays conservative and touches only artefacts that
are unmistakably Axiom Border's.
{{% /notice %}}

`AXIOM_RESTORE_NETPLAN=1` warns about possible connectivity loss. In practice an established SSH session
usually survives, since the TCP connection is not torn down and the IP is retained — but do not count on
it over a link you cannot recover from out-of-band.

## Troubleshooting by symptom

### Quick triage

| Symptom | Look here first |
| --- | --- |
| Service will not start | `journalctl -u <service> -n 50 --no-pager` |
| Service active but no results | `/opt/axiom-border/logs/axiom-border.log` |
| Metrics not reaching InfluxDB | `journalctl -u axiom-influxdb` plus the agent log |
| Monitoring node not reaching central | `tail -f /var/log/oda-lite/oda-lite.log \| grep "outputs.http"` |
| SSH, USB or interface events missing | The agent log, filtering by plugin name |
| Service restart-looping | `journalctl -u <service> --since "5 min ago"`, looking for `start-limit` |
| Install or upgrade behaved oddly | `tail -200 /var/log/axiom-border-install.log` |
| Disk filling up | `du -sh /var/log/oda-lite /opt/axiom-border/logs /var/log/journal/` |

### Detailed cases

**The service will not start.** Three usual causes: the binary lost its execute bit
(`chmod +x /opt/axiom-border/axiom-border`), a required port is occupied, or the YAML is invalid.
Validate the configuration directly:

```bash
python3 -c "import yaml; yaml.safe_load(open('/opt/axiom-border/config/configuration.yaml'))"
```

**Login fails with the correct password.** `login.pass` must be the **SHA-256 digest**, not the
plaintext. Recalculate and restart:

```bash
echo -n "<your-password>" | sha256sum | awk '{print $1}'
```

**A port is in use.** Identify the holder, then either stop it or move the Axiom Border port:

```bash
sudo ss -tlnp | grep -E ':(8083|1883|1888|8086|9090|9091) '
```

Ports 8083, 1883 and 1888 move via `apiPort`, `mqtt.broker.port` and `mqtt.broker.ws.port`. Ports 9090
and 9091 move in `/etc/oda-lite/oda-lite.conf`.

**Vulnerability scans return zero findings, suspiciously.** The web template directory is probably
empty or incomplete:

```bash
ls /opt/axiom-border/vulnscan-templates/ | head
```

It should list per-protocol directories such as `http/`, `network/` and `ssl/`. CVE templates live under
`http/cves/`, not at the root. Refresh them as described above. OT/ICS findings are unaffected, because
that suite ships with the product.

**Sniffing captures nothing.** Check the interface exists (`ip -o link show`), that the service runs as
root (`systemctl show axiom-border -p User`), and that there is actually traffic
(`sudo tcpdump -i <iface> -c 10`).

**The UI loads but every API call returns 401.** The JWT is missing or expired. Clear the browser's
`localStorage` and log in again.

**nmap probes hang or take very long.** A large target range with poor reachability, a high `timeout`,
or `udp: true` with many ports. Narrow the range, lower the timeout, disable UDP.

**High memory usage.** Check for an AI capability working a large dataset, an Influx bucket with no
retention policy, or `logLevel: DEBUG` left on.

**"Orphaned installation detected."** Bundle artefacts exist without an `.install-manifest`. Three ways
out: `AXIOM_CLEAN_RESIDUALS=1 sudo bash install.sh` moves the leftovers aside, `sudo bash upgrade.sh`
preserves data, `sudo bash uninstall.sh` cleans best-effort.

**`upgrade.sh` exits with code 2 reporting a rollback.** The new binary failed its health check and the
old one is back in service. Diagnose with the journal, the application log and the install log, then
retry with a corrected bundle.

**The monitoring node floods the journal with "connection refused" to port 9090.** The central node is
not up, a firewall blocks it, or the URL is wrong. The agent keeps retrying and buffers metrics while it
waits, so a short outage loses nothing — a long one eventually will.

**`Error executing 'netstat': exit status 1`.** Minimal Ubuntu Server images lack `net-tools`. Not
blocking — the other plugins continue:

```bash
sudo apt-get install -y net-tools && sudo systemctl restart oda-lite
```

**NetworkManager is active but interfaces show as `unmanaged`.** The netplan renderer was not switched
over. Re-run `install.sh --switch-to-network-manager`; it detects the current state and completes the
switch without reinstalling NetworkManager.

**`/etc/oda-lite/oda-lite.conf.new` contains literal `${VARIABLE}` placeholders.** The file is an
unexpanded template. **Do not copy it over the live configuration** — the agent would not start. Delete
it and re-run the upgrade with the current bundle.

**`scp` or `tar -xzf` fail with `Permission denied` in `/tmp`.** Some base images ship `/tmp` with wrong
permissions. It should be `drwxrwxrwt`:

```bash
ls -ld /tmp
sudo chmod 1777 /tmp
```