# How it works

## The whole loop in one picture

Everything in this section is a step of the loop below. The names in bold are the API resources you will meet
on the following pages.

```mermaid
sequenceDiagram
    participant U as You
    participant TA as Trainers API
    participant SC as Scheduler
    participant J as Training job
    participant IA as Inferencers API
    participant RU as Rules engine

    U->>TA: POST trainer (plan, data source, schedule)
    TA->>SC: schedule image execution or pipeline
    SC-->>J: at the scheduled time: export data, run the plan's image
    J->>J: train, evaluate, register the model version
    J->>J: build the inference image and push it
    J->>IA: create inferencer, or add the new version to it
    J->>RU: create the rule (first time only)
    J-->>SC: callback: OK
    U->>IA: activate a version
    IA->>RU: deploy the service, activate the rule
    RU->>IA: on each matching reading: POST prediction request
    IA-->>RU: prediction
    RU->>RU: collect datastreams, open or close alarms
```

## 1. You create a trainer

A **trainer** is a request to run a **training plan** on a data source, optionally on a schedule
(`POST /ai/organization/{organizationId}/trainer`, see [Trainers](../trainers/)). You give it:

- the **plan** to run, by its identifier from the [catalogue](../training_plans/);
- the **model name** — lowercase letters, digits and hyphens. It becomes the name of the model, of the
  inferencer, of the rule and of the scheduler entry, so choose it with care: it cannot be changed later and
  only one trainer per model name can exist in an organization;
- the plan's **configuration** — the fields the plan declares, such as the APN to learn;
- a **data source** — a file in your organization's file space, or a time series to export;
- a **schedule**, if the model has to be retrained periodically; and
- an execution **timeout** for the training job.

The Trainers API checks the plan exists, stores a small Kubernetes secret with your API key and organization so
the job can call the platform back on your behalf, and hands the work to the **scheduler**.

## 2. The scheduler runs it

The [scheduler](../scheduler/) is a general-purpose service: it runs REST requests, container images and pipelines
of both on a cron expression or an interval, and keeps a history of every execution. A trainer becomes one of
two things there, both named after the model:

| Data source | What is scheduled | Steps |
|---|---|---|
| A **file** (`source.path`) | An *image execution* | Run the plan's container with `dataSourcePath=/data/<your path>` |
| A **time series** (`source.timeserie`) | A *pipeline* | 1. `POST` the time series [Parquet export](../../data_retrieval/time_series/querying/#parquet-export), writing `<model>-<plan>-<timeseries>.parquet` into your file space<br>2. Run the plan's container with `dataSourcePath` pointing at that file |

Inside the job your organization's file space is mounted at **`/data`**, which is why every path the plan sees
starts there. The job also receives the plan's configuration as environment variables, the platform-wide AI
settings from a shared secret, and a `callbackUri` it must call when it finishes. The training job is a
Kubernetes Job with no retries: it either completes, fails, or is killed when the timeout expires.

## 3. The job trains, versions and publishes

Every plan image runs the same three commands, provided by the
[training template framework](../building_training_plans/):

1. **Generate the run profile** — experiment `<organization>-<model>`, registered model
   `<organization>-<model>-<algorithm>`, data location from `dataSourcePath`.
2. **Run the recipe** — ingest, split, transform, train, evaluate, register. Each run is tracked in the
   platform's MLflow, which is where model **versions** come from: the first successful training registers
   version 1, a retraining registers version 2, and so on. If the data does not meet the plan's minimum, the
   run fails here and says so.
3. **Publish the inference** — download the latest run's model and metrics, wrap them in the framework's
   FastAPI inference server, build a container image named `<organization>-<model>:v<version>` and push it to the
   platform registry. Then, through the [Inferencers API](../inferencers/):
   - if an inferencer named after the model **does not exist yet**, create it with that image as its only
     version — and, unless the trainer was created with `createRule: false`, first create the plan's **rule** in
     `default_channel` and link it to the inferencer;
   - if it **already exists**, add the image as a new version.

Whatever happens, the job reports back to the scheduler with `OK`, `ERROR` or — if it was killed by the timeout —
`TIMEOUT`, and a description. That report is what you see as the execution's **history**.

## 4. You activate a version

A freshly created inferencer has one version and **nothing deployed**. Activation is your call
(`PUT .../inferencer/{inferencerId}/activation?image=<version>&active=true`), because it is the moment the
platform starts spending resources on your behalf:

- the Inferencers API deploys the image as a service reachable **inside the platform** at
  `https://<organization>-<model>:8443/api/predict` — lowercase, underscores turned into hyphens;
- it waits for the container to be running;
- it sets the linked rules to `active: true`.

You can activate a specific version, or **`latest`**: then every new version a retraining produces replaces the
running one automatically, and the rule keeps calling the same address. Deactivating (`active=false`) undeploys
the service and deactivates the rules; the versions stay, ready to be activated again.

## 5. The rule scores every reading

The rule a plan creates is an ordinary [ADVANCED rule](../../management/organizations/channels/rules/): it
triggers on the datastream the plan cares about, builds the request the model expects, calls the inferencer with
`http.client`, and translates the answer into your data model — typically a boolean datastream saying whether the
reading was anomalous, a score, an explanation, and an alarm that opens when the entity turns anomalous and
closes when it returns to normal. The rule is created **inactive** and is switched on and off together with the
inferencer, so a deactivated model never leaves a rule calling a service that is not there.

The rule is yours: you can read it, tune its thresholds or change what it collects in the rules editor like any
other. Deleting the inferencer deletes its rules too, unless another inferencer still uses them.

## Naming, in one table

Once you know the organization name and the model name, you can predict every other name the feature creates:

| Thing | Name | Example for organization `acme`, model `radius-anomalies` |
|---|---|---|
| Scheduler entry (image execution or pipeline) | `<model>` | `radius-anomalies` |
| Exported time series file | `<model>-<plan id>-<time series id>.parquet` | `radius-anomalies-…-….parquet` |
| MLflow experiment | `<organization>-<model>` | `acme-radius-anomalies` |
| Registered model | `<organization>-<model>-<algorithm>` | `acme-radius-anomalies-isolation-forest` |
| Inference image | `<organization>-<model>:v<version>` | `acme-radius-anomalies:v3` |
| Inferencer | `<model>` | `radius-anomalies` |
| Deployed service | `<organization>-<model>` (lowercase, `_` → `-`) | `acme-radius-anomalies` |
| Inferencer endpoint | `https://<service>:8443/api/predict` | `https://acme-radius-anomalies:8443/api/predict` |
| Rule | `<model>`, in `default_channel` | `radius-anomalies` |

## Retraining

A trainer with a `schedule.expression` runs every time the expression fires, and each run adds a **new version**
to the same inferencer. The scheduler understands standard five-field cron and the extended form with a leading
seconds field and a trailing year field; the web console writes seven-field expressions such as
`0 0 0 15 */3 ? *` — midnight on the 15th, every third month. If the expression pins a single instant (every field
numeric, year included), the trainer is a one-off and the API reports `hasRetraining: false`.

A trainer created **without a schedule** runs once, about a minute after it is created.

An inferencer keeps a bounded number of versions (five by default). When a new one arrives over the limit, the
oldest **inactive** version is dropped; the active one is never removed by a retraining.

## Callbacks and history

Because trainings take minutes to hours, nothing in this loop blocks. The scheduler records every execution in
its **history** (`GET /scheduler/organization/{organizationId}/history`), with a state — `IN_PROGRESS` until the
callback arrives, `FINISHED` when it does, `FINISHED OUT OF TIME` if it arrived after the wait expired — and one
entry per step with its result and description. For a time-series trainer that is two steps: the export and the
training. The web console's *See history* action is a view of exactly this.

## What you need

- A user with the `root` or `super_admin_domain` profile: the five AI services accept no other.
- Authentication is the usual `X-ApiKey` header, or `Authorization: Bearer <JWT>`.
- The AI services are exposed on the same host as the rest of the OpenGate API, under the prefixes
  `/ai` (training plans, trainers, inferencers), `/scheduler` and `/fileConnector`.
