# AI Capabilities

## AI capabilities

Axiom Border can run machine-learning models **on the probe**, in containers, to detect anomalies in the
metrics it ingests. A model trains locally on local data, exposes an inference endpoint on loopback, and
rules call it to decide whether an event is worth an alarm.

The point is the same as everywhere else in the product: no cloud round trip, no data leaving the site.

{{% notice style="warning" title="Linux and the central role are required" icon="triangle-exclamation" %}}
This feature requires **Linux** and the container engine that the **`central`** role installs, so deploy
AI capabilities on that role.

Anywhere else the rest of the probe runs normally — every other subsystem is unaffected — but these
endpoints report that the container engine is unavailable:

`POST /iacap/{agentType}` · `GET /iacap/{agentType}/execute` · `DELETE /iacap/{agentType}` ·
`POST /containerd/loadImage` · `POST /containerd/deployImage`
{{% /notice %}}

{{% notice style="note" title="Anomaly detection covers system metrics" icon="lightbulb" %}}
Deploy the capability with agent type **`metrics`**, which detects anomalies in the system metrics the
probe ingests. The other agent types — `ssh`, `iface`, `usb` — appear in the API but are not supported
yet.
{{% /notice %}}

## Where you manage it

**AI capabilities** in the console lists the models configured on the probe. On a fresh installation it is
empty, because model images are supplied separately from the product:

{{< staticImage "edge_products/axiom_border/console-ai-capabilities.png" "The AI capabilities view before any model has been deployed" >}}

From here you create a capability, follow its training state, and read the quality metrics the model
reports about itself. **Create new AI trainer** opens the wizard that collects the same parameters as the
API call shown further down:

{{< staticImage "edge_products/axiom_border/wizard-ai-trainer.png" "The new AI trainer wizard" >}}

The equivalent API calls are shown alongside each step below.

## The lifecycle

```mermaid
flowchart TB
    TAR["Model image tarball<br>supplied separately"]:::ext -->|"stage on disk"| IMG["Image imported<br>for one agent type"]
    IMG -->|"POST /iacap"| EXPORT["Training data<br>exported"]
    EXPORT --> DEPLOY["Container deployed"]
    DEPLOY --> HEALTH{"Health check"}
    HEALTH -->|"not yet"| HEALTH
    HEALTH -->|"responds"| READY["status: ready"]
    READY -->|"retrain due"| TRAIN["status: training"]
    TRAIN --> READY
    READY -->|"predict"| INFER["Inference<br>from rules"]
    classDef ext fill:#e9edfa,stroke:#486ac9,color:#101010
```

Each agent type has a fixed loopback port, which is how a rule reaches the right model:

| Agent | Port | Container name |
| --- | --- | --- |
| `iface` | 5555 | `<trainerContainerName>-iface` |
| `ssh` | 5556 | `<trainerContainerName>-ssh` |
| `usb` | 5557 | `<trainerContainerName>-usb` |
| `metrics` | 5558 | `<trainerContainerName>-metrics` |

Each model listens on the host's loopback address, so rules reach it at `127.0.0.1:<port>`.

## Loading an image

Model images are **not** part of the installation bundle — they are distributed separately as OCI
tarballs, and `resources/iaimages/` ships empty as a placeholder.

Place the tarball in that directory, then import it:

```bash
# See what is on disk
curl -s http://192.168.1.10:8083/containerd/listdiskiaimages \
  -H "Authorization: Bearer <jwt-token>"

# Import it, bound to an agent type
curl -s -X POST 'http://192.168.1.10:8083/containerd/loadImage?agent=metrics&fileondisk=model-metrics.tar.gz' \
  -H "Authorization: Bearer <jwt-token>"
```

You can also upload the tarball directly as a multipart form field named `file`, without staging it on
disk first.

On import the image is tagged `axiom-border:<agent>`. Loading from the host with `nerdctl` works too, and
is the documented path in [Operation and maintenance](../operations/#loading-ai-images):

```bash
sudo nerdctl --namespace axiom-border image load -i model-metrics.tar.gz
```

{{% notice style="warning" title="Loading an image removes the other agents' images" icon="triangle-exclamation" %}}
`loadImage` keeps only the image for the agent you named and **deletes the other loaded images**. If you
intend to run capabilities for more than one agent type, be aware that importing sequentially will not
accumulate them.
{{% /notice %}}

## Deploying a capability

```bash
curl -s -X POST http://192.168.1.10:8083/iacap/metrics \
  -H "Authorization: Bearer <jwt-token>" \
  -H 'Content-Type: application/json' \
  -d '{
        "retrainFreqMinutes": 1440,
        "ruleEnabled": true
      }'
```

| Field | Type | Meaning |
| --- | --- | --- |
| `retrainFreqMinutes` | int64 | Retraining interval. **Minimum 10 minutes.** Zero means no retraining |
| `ruleEnabled` | bool | Enables the rule script for this agent |

Deployment does four things in order: exports the training data from the metrics database to a Parquet
file, deploys the container with that data available to it, waits for the container's health endpoint,
then marks the capability `ready`, collects the model metrics and enables the rule script.

A `retrainFreqMinutes` below 10 is rejected with `freq must be at least 10m0s`. A capability already
existing for that agent is rejected too — remove it first.

The health wait uses `trainerDetails.healthCheck`, which defaults to 20 attempts at 5 second intervals,
giving roughly 100 seconds for the container to come up. A slow-starting model may need that raised; see
[Configuration](../configuration/#trainerdetails).

## Checking status

```bash
curl -s http://192.168.1.10:8083/iacap \
  -H "Authorization: Bearer <jwt-token>"
```

Returns `204` when no capability is deployed. Otherwise, per capability: `iaName`, `enabled`, `status`,
`retraining`, `retrainFreqMinutes`, `lastExecution`, `nextExecution`, `fileFrom` (the tarball it came
from) and `metrics` — the model's own quality metrics, fetched from the container.

### Status values

| Status | Meaning |
| --- | --- |
| `ready` | Trained and serving inference |
| `training` | Training in progress |
| `ready (last training cancel, waiting next one)` | A training run was cancelled; the schedule continues |
| `canceled, waiting to resume` | Cancelled with no retraining scheduled |
| `error` | Something failed; details accompany the status |

## Training

Retraining runs automatically on the configured interval. To force a run:

```bash
curl -s http://192.168.1.10:8083/iacap/metrics/execute \
  -H "Authorization: Bearer <jwt-token>"
```

This is rejected while a training run is already in progress, and it waits for any scheduled run to
finish rather than colliding with it. An on-demand run updates the last-executed timestamp but **does not
shift the schedule** — the next scheduled run happens when it was always going to.

To cancel a run in progress:

```bash
curl -s http://192.168.1.10:8083/iacap/metrics/cancel \
  -H "Authorization: Bearer <jwt-token>"
```

Cancellation stops the health-check retry loop rather than killing the container mid-computation. The
capability lands in one of the two cancelled states above depending on whether retraining is scheduled.

Each training run re-exports the data, restarts the container, and waits for health again — so a training
run means a brief window where inference is unavailable. Rules calling `predict` during that window will
get an error, which they should handle rather than treating as an anomaly.

## Changing the schedule

```bash
curl -s -X PUT http://192.168.1.10:8083/iacap/metrics \
  -H "Authorization: Bearer <jwt-token>" \
  -H 'Content-Type: application/json' \
  -d '{"retrainFreqMinutes": 720, "ruleEnabled": true}'
```

Rejected while training is in progress. The same 10-minute minimum applies, and zero disables
retraining.

## Removing a capability

```bash
curl -s -X DELETE http://192.168.1.10:8083/iacap/metrics \
  -H "Authorization: Bearer <jwt-token>"
```

Best-effort cleanup: the container, the image, the rule script, the exported data directory and the
database record, plus stopping the retraining schedule. Note that this returns `200` even when parts of
the cleanup failed, with the errors in the response body — read it rather than assuming success.

## Calling inference from rules

This is where the capability earns its place. A rule script fetches recent metrics, builds a feature
vector, asks the model to score it, and raises an alarm when the model says anomaly:

```javascript
function process(metrics) {
  for (var i = 0; i < metrics.length; i++) {
    var metric = metrics[i];
    metric.SetT0();

    var payload = JSON.stringify({
      sent_vars:     metric.GetFieldByName("sent_vars"),
      prepared_vars: metric.GetFieldByName("prepared_vars"),
      ram_usage:     metric.GetFieldByName("ram_usage")
    });

    var response = newPredict(payload, "metrics");
    var result   = JSON.parse(response);

    if (result.prediction === 1) {
      newAlarm(metric, "critical", "New anomaly detected", "Anomaly", "AI Capability");
    }

    metric.SetT1();
  }
}
```

Two globals are available to scripts for this: `newPredict(payloadJSON, agentType)` calls the model's
inference endpoint, and `newGetMetrics(agentType)` fetches the model's own metrics. Both target the
loopback port for that agent, and both are configured under `trainerDetails`.

{{% notice style="warning" title="Keep the script time limit above the predict timeout" icon="triangle-exclamation" %}}
The `predict` call has its own timeout (default 10 s), but the **entire script invocation** is bounded by
`gojaTimeout` (default 8 s). With those two defaults, a slow inference call cannot complete — the script
is stopped first.

If your rules call inference, set `gojaTimeout` comfortably above `trainerDetails.predict.timeout`, or
lower the predict timeout. Always define `gojaTimeout`: without a time limit, scripts do not run at all.
{{% /notice %}}

The rule script for an agent is enabled through `ruleEnabled` when deploying the capability, and can be
managed directly through the `/scripts` endpoints. See [Alarms and rules](../alarms_and_rules/).

## Surviving a restart

Capabilities are part of the probe's local state, and startup restores them: containers are resumed,
model metrics are re-fetched, and retraining schedules resume from their stored next-run time.

One case is treated deliberately: a capability that was **`training` when the process stopped** is
considered cancelled rather than resumed, because the training run did not finish. It moves to the
appropriate cancelled state and waits for its next scheduled run.

Failures during this restoration are logged and do not prevent startup.

## Configuration reference

The relevant block is `trainerDetails` — container name, TLS material, and the three endpoint
definitions for health, metrics and inference. See
[Configuration](../configuration/#trainerdetails).

The health, metrics and inference endpoints are meant to be reached over loopback only. Keep them bound
to `127.0.0.1` and do not expose those ports beyond the host.