AI Capabilities

AI capabilities

Axiom Border can run machine-learning models on the probe, in containers, to detect anomalies in the metrics it ingests. A model trains locally on local data, exposes an inference endpoint on loopback, and rules call it to decide whether an event is worth an alarm.

The point is the same as everywhere else in the product: no cloud round trip, no data leaving the site.

Where you manage it

AI capabilities in the console lists the models configured on the probe. On a fresh installation it is empty, because model images are supplied separately from the product:

From here you create a capability, follow its training state, and read the quality metrics the model reports about itself. Create new AI trainer opens the wizard that collects the same parameters as the API call shown further down:

The equivalent API calls are shown alongside each step below.

The lifecycle

flowchart TB
    TAR["Model image tarball<br>supplied separately"]:::ext -->|"stage on disk"| IMG["Image imported<br>for one agent type"]
    IMG -->|"POST /iacap"| EXPORT["Training data<br>exported"]
    EXPORT --> DEPLOY["Container deployed"]
    DEPLOY --> HEALTH{"Health check"}
    HEALTH -->|"not yet"| HEALTH
    HEALTH -->|"responds"| READY["status: ready"]
    READY -->|"retrain due"| TRAIN["status: training"]
    TRAIN --> READY
    READY -->|"predict"| INFER["Inference<br>from rules"]
    classDef ext fill:#e9edfa,stroke:#486ac9,color:#101010

Each agent type has a fixed loopback port, which is how a rule reaches the right model:

Each model listens on the host’s loopback address, so rules reach it at 127.0.0.1:<port>.

Loading an image

Model images are not part of the installation bundle — they are distributed separately as OCI tarballs, and resources/iaimages/ ships empty as a placeholder.

Place the tarball in that directory, then import it:

# See what is on disk
curl -s http://192.168.1.10:8083/containerd/listdiskiaimages \
  -H "Authorization: Bearer <jwt-token>"

# Import it, bound to an agent type
curl -s -X POST 'http://192.168.1.10:8083/containerd/loadImage?agent=metrics&fileondisk=model-metrics.tar.gz' \
  -H "Authorization: Bearer <jwt-token>"

You can also upload the tarball directly as a multipart form field named file, without staging it on disk first.

On import the image is tagged axiom-border:<agent>. Loading from the host with nerdctl works too, and is the documented path in Operation and maintenance:

sudo nerdctl --namespace axiom-border image load -i model-metrics.tar.gz

Deploying a capability

curl -s -X POST http://192.168.1.10:8083/iacap/metrics \
  -H "Authorization: Bearer <jwt-token>" \
  -H 'Content-Type: application/json' \
  -d '{
        "retrainFreqMinutes": 1440,
        "ruleEnabled": true
      }'

Deployment does four things in order: exports the training data from the metrics database to a Parquet file, deploys the container with that data available to it, waits for the container’s health endpoint, then marks the capability ready, collects the model metrics and enables the rule script.

A retrainFreqMinutes below 10 is rejected with freq must be at least 10m0s. A capability already existing for that agent is rejected too — remove it first.

The health wait uses trainerDetails.healthCheck, which defaults to 20 attempts at 5 second intervals, giving roughly 100 seconds for the container to come up. A slow-starting model may need that raised; see Configuration.

Checking status

curl -s http://192.168.1.10:8083/iacap \
  -H "Authorization: Bearer <jwt-token>"

Returns 204 when no capability is deployed. Otherwise, per capability: iaName, enabled, status, retraining, retrainFreqMinutes, lastExecution, nextExecution, fileFrom (the tarball it came from) and metrics — the model’s own quality metrics, fetched from the container.

Status values

Training

Retraining runs automatically on the configured interval. To force a run:

curl -s http://192.168.1.10:8083/iacap/metrics/execute \
  -H "Authorization: Bearer <jwt-token>"

This is rejected while a training run is already in progress, and it waits for any scheduled run to finish rather than colliding with it. An on-demand run updates the last-executed timestamp but does not shift the schedule — the next scheduled run happens when it was always going to.

To cancel a run in progress:

curl -s http://192.168.1.10:8083/iacap/metrics/cancel \
  -H "Authorization: Bearer <jwt-token>"

Cancellation stops the health-check retry loop rather than killing the container mid-computation. The capability lands in one of the two cancelled states above depending on whether retraining is scheduled.

Each training run re-exports the data, restarts the container, and waits for health again — so a training run means a brief window where inference is unavailable. Rules calling predict during that window will get an error, which they should handle rather than treating as an anomaly.

Changing the schedule

curl -s -X PUT http://192.168.1.10:8083/iacap/metrics \
  -H "Authorization: Bearer <jwt-token>" \
  -H 'Content-Type: application/json' \
  -d '{"retrainFreqMinutes": 720, "ruleEnabled": true}'

Rejected while training is in progress. The same 10-minute minimum applies, and zero disables retraining.

Removing a capability

curl -s -X DELETE http://192.168.1.10:8083/iacap/metrics \
  -H "Authorization: Bearer <jwt-token>"

Best-effort cleanup: the container, the image, the rule script, the exported data directory and the database record, plus stopping the retraining schedule. Note that this returns 200 even when parts of the cleanup failed, with the errors in the response body — read it rather than assuming success.

Calling inference from rules

This is where the capability earns its place. A rule script fetches recent metrics, builds a feature vector, asks the model to score it, and raises an alarm when the model says anomaly:

function process(metrics) {
  for (var i = 0; i < metrics.length; i++) {
    var metric = metrics[i];
    metric.SetT0();

    var payload = JSON.stringify({
      sent_vars:     metric.GetFieldByName("sent_vars"),
      prepared_vars: metric.GetFieldByName("prepared_vars"),
      ram_usage:     metric.GetFieldByName("ram_usage")
    });

    var response = newPredict(payload, "metrics");
    var result   = JSON.parse(response);

    if (result.prediction === 1) {
      newAlarm(metric, "critical", "New anomaly detected", "Anomaly", "AI Capability");
    }

    metric.SetT1();
  }
}

Two globals are available to scripts for this: newPredict(payloadJSON, agentType) calls the model’s inference endpoint, and newGetMetrics(agentType) fetches the model’s own metrics. Both target the loopback port for that agent, and both are configured under trainerDetails.

The rule script for an agent is enabled through ruleEnabled when deploying the capability, and can be managed directly through the /scripts endpoints. See Alarms and rules.

Surviving a restart

Capabilities are part of the probe’s local state, and startup restores them: containers are resumed, model metrics are re-fetched, and retraining schedules resume from their stored next-run time.

One case is treated deliberately: a capability that was training when the process stopped is considered cancelled rather than resumed, because the training run did not finish. It moves to the appropriate cancelled state and waits for its next scheduled run.

Failures during this restoration are logged and do not prevent startup.

Configuration reference

The relevant block is trainerDetails — container name, TLS material, and the three endpoint definitions for health, metrics and inference. See Configuration.

The health, metrics and inference endpoints are meant to be reached over loopback only. Keep them bound to 127.0.0.1 and do not expose those ports beyond the host.